[jira] Updated: (ZOOKEEPER-436) Bookies should auto register to ZooKeeper
[ https://issues.apache.org/jira/browse/ZOOKEEPER-436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-436: Status: Patch Available (was: Open) Changes so the BookieServer's main method to create Bookie's will take in an additional parameter for the String of ZooKeeper servers (comma separated string of hostname:port). This is passed to the Bookie constructor which will invoke the ZK related client instantiation and creation of the ephemeral nodes for the Bookie Server being created. Bookies should auto register to ZooKeeper - Key: ZOOKEEPER-436 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-436 Project: Zookeeper Issue Type: Improvement Components: contrib-bookkeeper Reporter: Benjamin Reed Attachments: ZOOKEEPER-436.patch currently bookies have to be manually added to ZooKeeper to be used in a BookKeeper service. we should be able to just start up a bookkie, point it at ZooKeeper, and have it get auto integrated in. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-436) Bookies should auto register to ZooKeeper
[ https://issues.apache.org/jira/browse/ZOOKEEPER-436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-436: Status: Patch Available (was: Open) Bookies should auto register to ZooKeeper - Key: ZOOKEEPER-436 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-436 Project: Zookeeper Issue Type: Improvement Components: contrib-bookkeeper Reporter: Benjamin Reed Attachments: ZOOKEEPER-436.patch currently bookies have to be manually added to ZooKeeper to be used in a BookKeeper service. we should be able to just start up a bookkie, point it at ZooKeeper, and have it get auto integrated in. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-436) Bookies should auto register to ZooKeeper
[ https://issues.apache.org/jira/browse/ZOOKEEPER-436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-436: Attachment: ZOOKEEPER-436.patch Updated patch making the changes to fix the unit tests. Also made the changes without making the formatting changes done before. Bookies should auto register to ZooKeeper - Key: ZOOKEEPER-436 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-436 Project: Zookeeper Issue Type: Improvement Components: contrib-bookkeeper Reporter: Benjamin Reed Attachments: ZOOKEEPER-436.patch currently bookies have to be manually added to ZooKeeper to be used in a BookKeeper service. we should be able to just start up a bookkie, point it at ZooKeeper, and have it get auto integrated in. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-436) Bookies should auto register to ZooKeeper
[ https://issues.apache.org/jira/browse/ZOOKEEPER-436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-436: Status: Patch Available (was: Open) Bookies should auto register to ZooKeeper - Key: ZOOKEEPER-436 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-436 Project: Zookeeper Issue Type: Improvement Components: contrib-bookkeeper Reporter: Benjamin Reed Attachments: ZOOKEEPER-436.patch currently bookies have to be manually added to ZooKeeper to be used in a BookKeeper service. we should be able to just start up a bookkie, point it at ZooKeeper, and have it get auto integrated in. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-436) Bookies should auto register to ZooKeeper
[ https://issues.apache.org/jira/browse/ZOOKEEPER-436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12847007#action_12847007 ] Erwin Tam commented on ZOOKEEPER-436: - Oops, I knew i'd miss something from porting over the changes made earlier. Yeah, doing the formatting changes does make doing a quick glance at svn diff hard. Had this change on the original version but missed it when porting changes over manually. Thanks for catching it! Bookies should auto register to ZooKeeper - Key: ZOOKEEPER-436 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-436 Project: Zookeeper Issue Type: Improvement Components: contrib-bookkeeper Reporter: Benjamin Reed Fix For: 3.3.0 Attachments: ZOOKEEPER-436.patch, ZOOKEEPER-436.patch currently bookies have to be manually added to ZooKeeper to be used in a BookKeeper service. we should be able to just start up a bookkie, point it at ZooKeeper, and have it get auto integrated in. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-464) Need procedure to garbage collect ledgers
[ https://issues.apache.org/jira/browse/ZOOKEEPER-464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-464: Assignee: Erwin Tam I'm working on this enhancement now and just about ready to submit this as a patch. Will see if Ben wants to review my changes first beforehand. Need procedure to garbage collect ledgers - Key: ZOOKEEPER-464 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-464 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Erwin Tam An application using BookKeeper is likely to use a large number of ledgers over time. Such an application might not need all ledgers created over time and might want to delete some of these ledgers to free up some space on bookies. The idea of this jira is to implement a procedure that enables an application to garbage-collect unwanted ledgers. To garbage-collect a ledger, we need to delete the ledger metadata on ZooKeeper, and delete the ledger data on corresponding bookies. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-464) Need procedure to garbage collect ledgers
[ https://issues.apache.org/jira/browse/ZOOKEEPER-464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-464: Attachment: ZOOKEEPER-464.patch Uploaded a patch for BK ledger deletes. Could use more unit tests perhaps. Known issues are with using the BufferedChannel currently when reading in the ledgers that make up the entry log files. If the BufferedChannel is created for an already existing entry log file we want to read from (but not write to), it results in an infinite loop. Need procedure to garbage collect ledgers - Key: ZOOKEEPER-464 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-464 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Erwin Tam Attachments: ZOOKEEPER-464.patch An application using BookKeeper is likely to use a large number of ledgers over time. Such an application might not need all ledgers created over time and might want to delete some of these ledgers to free up some space on bookies. The idea of this jira is to implement a procedure that enables an application to garbage-collect unwanted ledgers. To garbage-collect a ledger, we need to delete the ledger metadata on ZooKeeper, and delete the ledger data on corresponding bookies. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-464) Need procedure to garbage collect ledgers
[ https://issues.apache.org/jira/browse/ZOOKEEPER-464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12849421#action_12849421 ] Erwin Tam commented on ZOOKEEPER-464: - Thanks for the feedback Flavio! 1. Where should I put the summary of the behavior? What about documentation of this feature? It does require changes to how you set up Bookie Servers (need to pass in the ZK servers list). Should I put that in comments for this Jira? Should I upload a text documentation too to describe the feature functionality? 2. I'll add the missing license text. Forgot that I need that for new files. 3. I have a fix for the BufferedChannel issue and it can be done either in the BufferedChannel code or outside it from the callers. The problem is when it is created on an existing File. The FileChannels passed to the constructor need to set their position to the end of the file. Otherwise, the write buffer's start position will default to 0, the start of the file. That screws things up when you try to read from it. This bug should already exist if you start up the Bookies with already existing entry logs and you try to read entries from there. I don't think there are any unit tests that test that specifically. Need procedure to garbage collect ledgers - Key: ZOOKEEPER-464 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-464 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Erwin Tam Fix For: 3.4.0 Attachments: ZOOKEEPER-464.patch An application using BookKeeper is likely to use a large number of ledgers over time. Such an application might not need all ledgers created over time and might want to delete some of these ledgers to free up some space on bookies. The idea of this jira is to implement a procedure that enables an application to garbage-collect unwanted ledgers. To garbage-collect a ledger, we need to delete the ledger metadata on ZooKeeper, and delete the ledger data on corresponding bookies. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-464) Need procedure to garbage collect ledgers
[ https://issues.apache.org/jira/browse/ZOOKEEPER-464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-464: Attachment: (was: ZOOKEEPER-464.patch) Need procedure to garbage collect ledgers - Key: ZOOKEEPER-464 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-464 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Erwin Tam Fix For: 3.4.0 An application using BookKeeper is likely to use a large number of ledgers over time. Such an application might not need all ledgers created over time and might want to delete some of these ledgers to free up some space on bookies. The idea of this jira is to implement a procedure that enables an application to garbage-collect unwanted ledgers. To garbage-collect a ledger, we need to delete the ledger metadata on ZooKeeper, and delete the ledger data on corresponding bookies. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-464) Need procedure to garbage collect ledgers
[ https://issues.apache.org/jira/browse/ZOOKEEPER-464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852527#action_12852527 ] Erwin Tam commented on ZOOKEEPER-464: - Thanks Flavio for the feedback! I'll make the changes you suggested and will upload a new patch (without deleting the old one). I'll bug Ben about the forrest xml documentation if I can't figure it out. Need procedure to garbage collect ledgers - Key: ZOOKEEPER-464 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-464 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Erwin Tam Fix For: 3.4.0 Attachments: ZOOKEEPER-464.patch An application using BookKeeper is likely to use a large number of ledgers over time. Such an application might not need all ledgers created over time and might want to delete some of these ledgers to free up some space on bookies. The idea of this jira is to implement a procedure that enables an application to garbage-collect unwanted ledgers. To garbage-collect a ledger, we need to delete the ledger metadata on ZooKeeper, and delete the ledger data on corresponding bookies. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-464) Need procedure to garbage collect ledgers
[ https://issues.apache.org/jira/browse/ZOOKEEPER-464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-464: Status: Open (was: Patch Available) Need procedure to garbage collect ledgers - Key: ZOOKEEPER-464 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-464 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Erwin Tam Fix For: 3.4.0 Attachments: zookeeper-464-log.txt, ZOOKEEPER-464.patch, ZOOKEEPER-464.patch An application using BookKeeper is likely to use a large number of ledgers over time. Such an application might not need all ledgers created over time and might want to delete some of these ledgers to free up some space on bookies. The idea of this jira is to implement a procedure that enables an application to garbage-collect unwanted ledgers. To garbage-collect a ledger, we need to delete the ledger metadata on ZooKeeper, and delete the ledger data on corresponding bookies. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-464) Need procedure to garbage collect ledgers
[ https://issues.apache.org/jira/browse/ZOOKEEPER-464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-464: Attachment: ZOOKEEPER-464.patch Updated patch with documentation changes and fix so the unit tests for ledger deletes won't error out intermittently. The problem was a bug in the EntryLogger where the BKLO ByteBuffer header that is prepended to each entry log was defined as a static object. If multiple Bookies within the JVM are running, a race condition can occur if they are rolling over entry logs and accessing this header at the same time. The end result is, some entry logs are created without the header and that screws the entry log ledger extraction. All new methods should have proper javadoc comments before them now too. Need procedure to garbage collect ledgers - Key: ZOOKEEPER-464 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-464 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Erwin Tam Fix For: 3.4.0 Attachments: zookeeper-464-log.txt, ZOOKEEPER-464.patch, ZOOKEEPER-464.patch, ZOOKEEPER-464.patch An application using BookKeeper is likely to use a large number of ledgers over time. Such an application might not need all ledgers created over time and might want to delete some of these ledgers to free up some space on bookies. The idea of this jira is to implement a procedure that enables an application to garbage-collect unwanted ledgers. To garbage-collect a ledger, we need to delete the ledger metadata on ZooKeeper, and delete the ledger data on corresponding bookies. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-464) Need procedure to garbage collect ledgers
[ https://issues.apache.org/jira/browse/ZOOKEEPER-464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-464: Status: Patch Available (was: Open) Latest version of the patch with documentation changes and fixes for some intermittent junit test errors. Need procedure to garbage collect ledgers - Key: ZOOKEEPER-464 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-464 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Erwin Tam Fix For: 3.4.0 Attachments: zookeeper-464-log.txt, ZOOKEEPER-464.patch, ZOOKEEPER-464.patch, ZOOKEEPER-464.patch An application using BookKeeper is likely to use a large number of ledgers over time. Such an application might not need all ledgers created over time and might want to delete some of these ledgers to free up some space on bookies. The idea of this jira is to implement a procedure that enables an application to garbage-collect unwanted ledgers. To garbage-collect a ledger, we need to delete the ledger metadata on ZooKeeper, and delete the ledger data on corresponding bookies. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-464) Need procedure to garbage collect ledgers
[ https://issues.apache.org/jira/browse/ZOOKEEPER-464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12855158#action_12855158 ] Erwin Tam commented on ZOOKEEPER-464: - Flavio, my comments a couple of entries above talks about the EntryLogger's BKLO header that was causing the problem. That was creating the issue in the junit test making it fail intermittently. I sent you and Ben an email about this fix too in more detail but we could talk about it more if you want. Thanks! Need procedure to garbage collect ledgers - Key: ZOOKEEPER-464 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-464 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Erwin Tam Fix For: 3.4.0 Attachments: zookeeper-464-log.txt, ZOOKEEPER-464.patch, ZOOKEEPER-464.patch, ZOOKEEPER-464.patch An application using BookKeeper is likely to use a large number of ledgers over time. Such an application might not need all ledgers created over time and might want to delete some of these ledgers to free up some space on bookies. The idea of this jira is to implement a procedure that enables an application to garbage-collect unwanted ledgers. To garbage-collect a ledger, we need to delete the ledger metadata on ZooKeeper, and delete the ledger data on corresponding bookies. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-464) Need procedure to garbage collect ledgers
[ https://issues.apache.org/jira/browse/ZOOKEEPER-464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-464: Status: Open (was: Patch Available) Need procedure to garbage collect ledgers - Key: ZOOKEEPER-464 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-464 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Erwin Tam Fix For: 3.4.0 Attachments: zookeeper-464-log.txt, ZOOKEEPER-464.patch, ZOOKEEPER-464.patch, ZOOKEEPER-464.patch, ZOOKEEPER-464.patch An application using BookKeeper is likely to use a large number of ledgers over time. Such an application might not need all ledgers created over time and might want to delete some of these ledgers to free up some space on bookies. The idea of this jira is to implement a procedure that enables an application to garbage-collect unwanted ledgers. To garbage-collect a ledger, we need to delete the ledger metadata on ZooKeeper, and delete the ledger data on corresponding bookies. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-464) Need procedure to garbage collect ledgers
[ https://issues.apache.org/jira/browse/ZOOKEEPER-464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-464: Attachment: ZOOKEEPER-464.patch Updated patch with formatting changes requested. Need procedure to garbage collect ledgers - Key: ZOOKEEPER-464 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-464 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Erwin Tam Fix For: 3.4.0 Attachments: zookeeper-464-log.txt, ZOOKEEPER-464.patch, ZOOKEEPER-464.patch, ZOOKEEPER-464.patch, ZOOKEEPER-464.patch An application using BookKeeper is likely to use a large number of ledgers over time. Such an application might not need all ledgers created over time and might want to delete some of these ledgers to free up some space on bookies. The idea of this jira is to implement a procedure that enables an application to garbage-collect unwanted ledgers. To garbage-collect a ledger, we need to delete the ledger metadata on ZooKeeper, and delete the ledger data on corresponding bookies. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-464) Need procedure to garbage collect ledgers
[ https://issues.apache.org/jira/browse/ZOOKEEPER-464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-464: Status: Patch Available (was: Open) Uploaded a new patch with the formatting changes. Note that the generated pdf's previously already had the issue with a line going beyond the paper margin. I modified this but let me know if it isn't done correctly. Need procedure to garbage collect ledgers - Key: ZOOKEEPER-464 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-464 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Erwin Tam Fix For: 3.4.0 Attachments: zookeeper-464-log.txt, ZOOKEEPER-464.patch, ZOOKEEPER-464.patch, ZOOKEEPER-464.patch, ZOOKEEPER-464.patch An application using BookKeeper is likely to use a large number of ledgers over time. Such an application might not need all ledgers created over time and might want to delete some of these ledgers to free up some space on bookies. The idea of this jira is to implement a procedure that enables an application to garbage-collect unwanted ledgers. To garbage-collect a ledger, we need to delete the ledger metadata on ZooKeeper, and delete the ledger data on corresponding bookies. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (ZOOKEEPER-712) Bookie recovery
[ https://issues.apache.org/jira/browse/ZOOKEEPER-712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam reassigned ZOOKEEPER-712: --- Assignee: Erwin Tam Bookie recovery --- Key: ZOOKEEPER-712 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-712 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Erwin Tam Recover the ledger fragments of a bookie once it crashes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-712) Bookie recovery
[ https://issues.apache.org/jira/browse/ZOOKEEPER-712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-712: Attachment: ZOOKEEPER-712.patch Bookie recovery --- Key: ZOOKEEPER-712 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-712 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Erwin Tam Attachments: ZOOKEEPER-712.patch Recover the ledger fragments of a bookie once it crashes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-712) Bookie recovery
[ https://issues.apache.org/jira/browse/ZOOKEEPER-712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-712: Status: Patch Available (was: Open) First pass at implementing bookie recovery as an admin tool. The patch uploaded has two new files: BookieRecoveryTest.java and BookKeeperTools.java (with a corresponding new directory under bookkeeper/tools). The main API that BookKeeperTools.java implements is the following: public void asyncRecoverBookieData(final InetSocketAddress bookieSrc, final InetSocketAddress bookieDest, final RecoverCallback cb, final Object context); The synchronous version of this API is here: public void recoverBookieData(final InetSocketAddress bookieSrc, final InetSocketAddress bookieDest) throws InterruptedException; This is to recover all of the bookie ledger data that was present on a dead input bookieSrc. The other input 'bookieDest' is optional and if passed, we will recover all data to that bookie server specifically. Otherwise for each ledger being recovered that was stored on the bookieSrc, we choose one of the other available bookie servers and re-replicate the ledger fragment entries to there. A command line way to invoke this is done via a main method in BookKeeperTools which expects the following 2-3 input parameters. USAGE: BookKeeperTools zkServers bookieSrc [bookieDest] zkServers is a comma separated list of host:port pairs for the ZooKeeper servers in the cluster. bookieSrc is the host:port for the bookie server we are recovering data from. bookieDest is the host:port (optional) for the bookie server we want to recover the data to. Current limitations: There is no way to know what each ledger's digest type or key/password are. That metadata isn't stored anywhere we can access readily. Thus for now, we are assuming that all ledgers to be recovered have been created with the same digest type and password. These values are set via java system properties called digestType and passwd. Once we have a way to store and retrieve this information (via ZK most likely and stored by the Bookie servers when new ledgers are created), we can modify this aspect of the bookie recovery tool. Pseudocode: Inputs: zkServers - Used to create a ZK client so we can read in all of the BK metadata needed to perform the bookie recovery. bookieSrc - Used to match against ledger metadata indicating which bookie servers comprise the ensembles that make up a ledger. bookieDest - Optionally used to write the recovered ledger data to directly. 1. Sync with ZK 2. Read from ZK to get all available bookie servers (only if bookieDest was not passed). 3. Read from ZK to get all active ledgers. 4. For each ledger, open it to obtain the LedgerHandle. 5. For each ledger fragment, see if the dead input bookieSrc is a part of the ensemble for it. 6. For each ledger fragment that needs to be recovered, find the entries that were actually stored on the bookieSrc. Since we stripe data across bookies in the ensemble, not all ledger entries in the fragment would have been stored on the bookieSrc. 7. For each of the ledger entries in the fragment that were stored on the bookieSrc, use the BookKeeper client to read it. Using the client will take care of choosing one of the other bookies where this data is available since the bookieSrc server is dead. 8. Choosing a new bookieDest (either passed in explicitly or chosen at random currently among the available bookie servers), write this ledger entry directly to there using the BookieClient (the layer that talks to the bookie servers directly). 9. Once all ledger entries for all fragments for a ledger have been recovered and re-replicated, update ZK so the ledger's metadata now points to the new bookie server where this data has been re-replicated to. We choose a single new bookie for each ledger to store the recovered data. Bookie recovery --- Key: ZOOKEEPER-712 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-712 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Erwin Tam Attachments: ZOOKEEPER-712.patch Recover the ledger fragments of a bookie once it crashes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-775) A large scale pub/sub system
[ https://issues.apache.org/jira/browse/ZOOKEEPER-775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-775: Attachment: ZOOKEEPER-775.patch Latest Hedwig apache branch code along with some simplified Hedwig server hub start up scripts. A large scale pub/sub system Key: ZOOKEEPER-775 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-775 Project: Zookeeper Issue Type: New Feature Components: contrib Reporter: Benjamin Reed Assignee: Benjamin Reed Fix For: 3.4.0 Attachments: libs.zip, libs_2.zip, ZOOKEEPER-775.patch, ZOOKEEPER-775.patch, ZOOKEEPER-775_2.patch, ZOOKEEPER-775_3.patch we have developed a large scale pub/sub system based on ZooKeeper and BookKeeper. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-775) A large scale pub/sub system
[ https://issues.apache.org/jira/browse/ZOOKEEPER-775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-775: Status: Patch Available (was: Open) Latest version of the Hedwig code in the apache branch. This also includes the C++ client bindings that Ivan Kelly worked on. Looking at past comments, if you are using the hw.bash script to startup ZK, BK, or Hedwig hubs, this is intended to be done on remote boxes. So you'll need to build/compile to get the Hedwig fat server jar and prepend the push_jar=true command to the hw.bash line so it'll copy over the jar to the remote server(s) and place it in the appropriate location. The CLASSPATH remotely is set to point to this area and won't work without it. A large scale pub/sub system Key: ZOOKEEPER-775 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-775 Project: Zookeeper Issue Type: New Feature Components: contrib Reporter: Benjamin Reed Assignee: Benjamin Reed Fix For: 3.4.0 Attachments: libs.zip, libs_2.zip, ZOOKEEPER-775.patch, ZOOKEEPER-775.patch, ZOOKEEPER-775_2.patch, ZOOKEEPER-775_3.patch we have developed a large scale pub/sub system based on ZooKeeper and BookKeeper. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-775) A large scale pub/sub system
[ https://issues.apache.org/jira/browse/ZOOKEEPER-775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-775: Status: Open (was: Patch Available) A large scale pub/sub system Key: ZOOKEEPER-775 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-775 Project: Zookeeper Issue Type: New Feature Components: contrib Reporter: Benjamin Reed Assignee: Benjamin Reed Fix For: 3.4.0 Attachments: libs.zip, libs_2.zip, ZOOKEEPER-775.patch, ZOOKEEPER-775.patch, ZOOKEEPER-775_2.patch, ZOOKEEPER-775_3.patch we have developed a large scale pub/sub system based on ZooKeeper and BookKeeper. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-775) A large scale pub/sub system
[ https://issues.apache.org/jira/browse/ZOOKEEPER-775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-775: Attachment: ZOOKEEPER-775.patch Newer patch removing the @author tags in the java classes and adding Apache license headers to the ones that need them. Hudson complained about release audit warnings for some text files but we don't usually add Apache license headers to these type of files. Will see if the autobuild will complain about that with the newer patch. A large scale pub/sub system Key: ZOOKEEPER-775 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-775 Project: Zookeeper Issue Type: New Feature Components: contrib Reporter: Benjamin Reed Assignee: Benjamin Reed Fix For: 3.4.0 Attachments: libs.zip, libs_2.zip, ZOOKEEPER-775.patch, ZOOKEEPER-775.patch, ZOOKEEPER-775.patch, ZOOKEEPER-775_2.patch, ZOOKEEPER-775_3.patch we have developed a large scale pub/sub system based on ZooKeeper and BookKeeper. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-775) A large scale pub/sub system
[ https://issues.apache.org/jira/browse/ZOOKEEPER-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12900093#action_12900093 ] Erwin Tam commented on ZOOKEEPER-775: - Mahadev, we've been using the internal hedwig/apache svn branch for making changes, at least until things are committed to Apache's SVN. You can grab the latest changes there and incorporate them perhaps with your modifications. http://svn.corp.yahoo.com/view/yahoo/yrl/hedwig/apache/ Ivan is correct, the p12.pass file is just a dummy password file used for testing SSL connections between client and server. We need that otherwise the unit tests would fail. A large scale pub/sub system Key: ZOOKEEPER-775 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-775 Project: Zookeeper Issue Type: New Feature Components: contrib Reporter: Benjamin Reed Assignee: Benjamin Reed Fix For: 3.4.0 Attachments: libs.zip, libs_2.zip, ZOOKEEPER-775.patch, ZOOKEEPER-775.patch, ZOOKEEPER-775.patch, ZOOKEEPER-775_2.patch, ZOOKEEPER-775_3.patch we have developed a large scale pub/sub system based on ZooKeeper and BookKeeper. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (ZOOKEEPER-861) Missing the test SSL certificate used for running junit tests.
Missing the test SSL certificate used for running junit tests. -- Key: ZOOKEEPER-861 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-861 Project: Zookeeper Issue Type: Bug Components: contrib-hedwig Reporter: Erwin Tam Assignee: Erwin Tam Priority: Minor Fix For: 3.4.0 The Hedwig code checked into Apache is missing a test SSL certificate file used for running the server junit tests. We need this file otherwise the tests that use this (e.g. TestHedwigHub) will fail. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (ZOOKEEPER-862) Hedwig created ledgers with hardcoded Bookkeeper ensemble and quorum size. Make these a server config parameter instead.
Hedwig created ledgers with hardcoded Bookkeeper ensemble and quorum size. Make these a server config parameter instead. - Key: ZOOKEEPER-862 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-862 Project: Zookeeper Issue Type: Improvement Components: contrib-hedwig Reporter: Erwin Tam Assignee: Erwin Tam Fix For: 3.4.0 Hedwig code right now when using Bookkeeper as the persistence store is hardcoding the number of bookie servers in the ensemble and quorum size. This is used the first time a ledger is created. This should be exposed as a server configuration parameter instead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-861) Missing the test SSL certificate used for running junit tests.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-861: Attachment: server.p12 Uploading the binary SSL certificate file used for doing junit tests. Missing the test SSL certificate used for running junit tests. -- Key: ZOOKEEPER-861 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-861 Project: Zookeeper Issue Type: Bug Components: contrib-hedwig Reporter: Erwin Tam Assignee: Erwin Tam Priority: Minor Fix For: 3.4.0 Attachments: server.p12 The Hedwig code checked into Apache is missing a test SSL certificate file used for running the server junit tests. We need this file otherwise the tests that use this (e.g. TestHedwigHub) will fail. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-861) Missing the test SSL certificate used for running junit tests.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-861: Status: Patch Available (was: Open) Missing the test SSL certificate used for running junit tests. -- Key: ZOOKEEPER-861 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-861 Project: Zookeeper Issue Type: Bug Components: contrib-hedwig Reporter: Erwin Tam Assignee: Erwin Tam Priority: Minor Fix For: 3.4.0 Attachments: server.p12, ZOOKEEPER-861.patch The Hedwig code checked into Apache is missing a test SSL certificate file used for running the server junit tests. We need this file otherwise the tests that use this (e.g. TestHedwigHub) will fail. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-862) Hedwig created ledgers with hardcoded Bookkeeper ensemble and quorum size. Make these a server config parameter instead.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-862: Status: Patch Available (was: Open) Fix so the bookkeeper ledgers created will use a server configuration parameter to determine the ensemble and quorum size instead of hardcoding it. Hedwig created ledgers with hardcoded Bookkeeper ensemble and quorum size. Make these a server config parameter instead. - Key: ZOOKEEPER-862 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-862 Project: Zookeeper Issue Type: Improvement Components: contrib-hedwig Reporter: Erwin Tam Assignee: Erwin Tam Fix For: 3.4.0 Attachments: ZOOKEEPER-862.patch Hedwig code right now when using Bookkeeper as the persistence store is hardcoding the number of bookie servers in the ensemble and quorum size. This is used the first time a ledger is created. This should be exposed as a server configuration parameter instead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-862) Hedwig created ledgers with hardcoded Bookkeeper ensemble and quorum size. Make these a server config parameter instead.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-862: Attachment: ZOOKEEPER-862.patch Hedwig created ledgers with hardcoded Bookkeeper ensemble and quorum size. Make these a server config parameter instead. - Key: ZOOKEEPER-862 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-862 Project: Zookeeper Issue Type: Improvement Components: contrib-hedwig Reporter: Erwin Tam Assignee: Erwin Tam Fix For: 3.4.0 Attachments: ZOOKEEPER-862.patch Hedwig code right now when using Bookkeeper as the persistence store is hardcoding the number of bookie servers in the ensemble and quorum size. This is used the first time a ledger is created. This should be exposed as a server configuration parameter instead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-861) Missing the test SSL certificate used for running junit tests.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12906283#action_12906283 ] Erwin Tam commented on ZOOKEEPER-861: - The patch won't really apply since it is just showing where the binary file server.p12 should be placed. I don't think there's a way for patches to work when adding new binary files to the source. Missing the test SSL certificate used for running junit tests. -- Key: ZOOKEEPER-861 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-861 Project: Zookeeper Issue Type: Bug Components: contrib-hedwig Reporter: Erwin Tam Assignee: Erwin Tam Priority: Minor Fix For: 3.4.0 Attachments: server.p12, ZOOKEEPER-861.patch The Hedwig code checked into Apache is missing a test SSL certificate file used for running the server junit tests. We need this file otherwise the tests that use this (e.g. TestHedwigHub) will fail. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-864) Hedwig C++ client improvements
[ https://issues.apache.org/jira/browse/ZOOKEEPER-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12907785#action_12907785 ] Erwin Tam commented on ZOOKEEPER-864: - Ivan, the changes look good to me. One question though, do we need to include the auto-generated protocol.h header to the Makefile.am? I had to do that step manually a couple of weeks ago otherwise things didn't work when using the hedwig client libs generated. Hedwig C++ client improvements -- Key: ZOOKEEPER-864 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-864 Project: Zookeeper Issue Type: Improvement Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: 3.4.0 Attachments: warnings.txt, ZOOKEEPER-864.diff, ZOOKEEPER-864.diff, ZOOKEEPER-864.diff I changed the socket code to use boost asio. Now the client only creates one thread, and all operations are non-blocking. Tests are now automated, just run make check. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-864) Hedwig C++ client improvements
[ https://issues.apache.org/jira/browse/ZOOKEEPER-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12909002#action_12909002 ] Erwin Tam commented on ZOOKEEPER-864: - +1 Latest patch looks good to me. We can just approve it and push it in first. Thanks for the great work Ivan! Hedwig C++ client improvements -- Key: ZOOKEEPER-864 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-864 Project: Zookeeper Issue Type: Improvement Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: 3.4.0 Attachments: warnings.txt, ZOOKEEPER-864.diff, ZOOKEEPER-864.diff, ZOOKEEPER-864.diff I changed the socket code to use boost asio. Now the client only creates one thread, and all operations are non-blocking. Tests are now automated, just run make check. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (ZOOKEEPER-886) Hedwig Server stays in disconnected state when connection to ZK dies but gets reconnected
Hedwig Server stays in disconnected state when connection to ZK dies but gets reconnected --- Key: ZOOKEEPER-886 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-886 Project: Zookeeper Issue Type: Bug Components: contrib-hedwig Reporter: Erwin Tam Assignee: Erwin Tam The Hedwig Server is connected to ZooKeeper. In the ZkTopicManager, it registers a watcher so that if it ever gets disconnected from ZK, it will temporarily fail all incoming requests since the Hedwig server does not know for sure if it is still the master for the topics. When the ZK client gets reconnected, the logic currently is wrong and it does not unset the suspended flag. Thus once it gets disconnected, it will stay in the suspended state forever, thereby making the Hedwig server hub dead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-886) Hedwig Server stays in disconnected state when connection to ZK dies but gets reconnected
[ https://issues.apache.org/jira/browse/ZOOKEEPER-886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-886: Attachment: ZOOKEEPER-886.patch Hedwig Server stays in disconnected state when connection to ZK dies but gets reconnected --- Key: ZOOKEEPER-886 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-886 Project: Zookeeper Issue Type: Bug Components: contrib-hedwig Reporter: Erwin Tam Assignee: Erwin Tam Attachments: ZOOKEEPER-886.patch The Hedwig Server is connected to ZooKeeper. In the ZkTopicManager, it registers a watcher so that if it ever gets disconnected from ZK, it will temporarily fail all incoming requests since the Hedwig server does not know for sure if it is still the master for the topics. When the ZK client gets reconnected, the logic currently is wrong and it does not unset the suspended flag. Thus once it gets disconnected, it will stay in the suspended state forever, thereby making the Hedwig server hub dead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-886) Hedwig Server stays in disconnected state when connection to ZK dies but gets reconnected
[ https://issues.apache.org/jira/browse/ZOOKEEPER-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12918178#action_12918178 ] Erwin Tam commented on ZOOKEEPER-886: - Bug was an error in the logic of the if condition. For the ZK reconnect condition, the if logic would bypass the part to change the server back to the not suspended state. Modified this so it will do it correctly now. Hedwig Server stays in disconnected state when connection to ZK dies but gets reconnected --- Key: ZOOKEEPER-886 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-886 Project: Zookeeper Issue Type: Bug Components: contrib-hedwig Reporter: Erwin Tam Assignee: Erwin Tam Attachments: ZOOKEEPER-886.patch The Hedwig Server is connected to ZooKeeper. In the ZkTopicManager, it registers a watcher so that if it ever gets disconnected from ZK, it will temporarily fail all incoming requests since the Hedwig server does not know for sure if it is still the master for the topics. When the ZK client gets reconnected, the logic currently is wrong and it does not unset the suspended flag. Thus once it gets disconnected, it will stay in the suspended state forever, thereby making the Hedwig server hub dead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-886) Hedwig Server stays in disconnected state when connection to ZK dies but gets reconnected
[ https://issues.apache.org/jira/browse/ZOOKEEPER-886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erwin Tam updated ZOOKEEPER-886: Status: Patch Available (was: Open) Uploaded patch for this fix. Hedwig Server stays in disconnected state when connection to ZK dies but gets reconnected --- Key: ZOOKEEPER-886 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-886 Project: Zookeeper Issue Type: Bug Components: contrib-hedwig Reporter: Erwin Tam Assignee: Erwin Tam Attachments: ZOOKEEPER-886.patch The Hedwig Server is connected to ZooKeeper. In the ZkTopicManager, it registers a watcher so that if it ever gets disconnected from ZK, it will temporarily fail all incoming requests since the Hedwig server does not know for sure if it is still the master for the topics. When the ZK client gets reconnected, the logic currently is wrong and it does not unset the suspended flag. Thus once it gets disconnected, it will stay in the suspended state forever, thereby making the Hedwig server hub dead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (ZOOKEEPER-931) Documentation for Hedwig
Documentation for Hedwig Key: ZOOKEEPER-931 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-931 Project: Zookeeper Issue Type: Task Components: contrib-hedwig Reporter: Erwin Tam This is a tracking jira for providing documentation to Hedwig on the Apache site. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
