[jira] Commented: (ZOOKEEPER-462) Last hint for open ledger
[ https://issues.apache.org/jira/browse/ZOOKEEPER-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12764566#action_12764566 ] Utkarsh Srivastava commented on ZOOKEEPER-462: -- The (new) client code is pretty much structured that way. Determining the last entry-id (recovery process) is separate from the process of opening the ledger. Last hint for open ledger - Key: ZOOKEEPER-462 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-462 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Flavio Paiva Junqueira Fix For: 3.3.0 Attachments: ZOOKEEPER-462.patch In some use cases of BookKeeper, it is useful to be able to read from a ledger before closing the ledger. To enable such a feature, the writer has to be able to communicate to a reader how many entries it has been able to write successfully. The main idea of this jira is to continuously update a znode with the number of successful writes, and a reader can, for example, watch the node for changes. I was thinking of having a configuration parameter to state how often a writer should update the hint on ZooKeeper (e.g., every 1000 requests, every 10,000 requests). Clearly updating more often increases the overhead of writing to ZooKeeper, although the impact on the performance of writes to BookKeeper should be minimal given that we make an asynchronous call to update the hint. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-462) Last hint for open ledger
[ https://issues.apache.org/jira/browse/ZOOKEEPER-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12764535#action_12764535 ] Benjamin Reed commented on ZOOKEEPER-462: - if the client asks all the bookies he may not be able to get the last committed entry if we allow for failures. the safest thing to do would be to get the last entry from each bookie and then use the entry id in the last committed field. that would mean that you would never be able to see the actual last committed record. i think it would be good to allow the client to specify the last committed entry on the open. that way we allow the client to figure out the last committed record any way it wants, via communication from other processes for example, and it would keep the open code simple: it would just use the id it wouldn't need to worry about recovery. Last hint for open ledger - Key: ZOOKEEPER-462 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-462 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Flavio Paiva Junqueira Fix For: 3.3.0 Attachments: ZOOKEEPER-462.patch In some use cases of BookKeeper, it is useful to be able to read from a ledger before closing the ledger. To enable such a feature, the writer has to be able to communicate to a reader how many entries it has been able to write successfully. The main idea of this jira is to continuously update a znode with the number of successful writes, and a reader can, for example, watch the node for changes. I was thinking of having a configuration parameter to state how often a writer should update the hint on ZooKeeper (e.g., every 1000 requests, every 10,000 requests). Clearly updating more often increases the overhead of writing to ZooKeeper, although the impact on the performance of writes to BookKeeper should be minimal given that we make an asynchronous call to update the hint. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-462) Last hint for open ledger
[ https://issues.apache.org/jira/browse/ZOOKEEPER-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12763572#action_12763572 ] Patrick Hunt commented on ZOOKEEPER-462: What's the status on this? Last hint for open ledger - Key: ZOOKEEPER-462 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-462 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Flavio Paiva Junqueira Fix For: 3.3.0 Attachments: ZOOKEEPER-462.patch In some use cases of BookKeeper, it is useful to be able to read from a ledger before closing the ledger. To enable such a feature, the writer has to be able to communicate to a reader how many entries it has been able to write successfully. The main idea of this jira is to continuously update a znode with the number of successful writes, and a reader can, for example, watch the node for changes. I was thinking of having a configuration parameter to state how often a writer should update the hint on ZooKeeper (e.g., every 1000 requests, every 10,000 requests). Clearly updating more often increases the overhead of writing to ZooKeeper, although the impact on the performance of writes to BookKeeper should be minimal given that we make an asynchronous call to update the hint. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-462) Last hint for open ledger
[ https://issues.apache.org/jira/browse/ZOOKEEPER-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12763595#action_12763595 ] Utkarsh Srivastava commented on ZOOKEEPER-462: -- Ben and I discussed this and we dont think this is the best design. Under the current design, a lot of unnecessary write load will be put on ZK. Instead, the bookies already support a method to query for the last entry for a particular ledger. Thus, a client that wants to read an unclosed ledger can ask the bookies for their last entries and read until there. Last hint for open ledger - Key: ZOOKEEPER-462 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-462 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Flavio Paiva Junqueira Fix For: 3.3.0 Attachments: ZOOKEEPER-462.patch In some use cases of BookKeeper, it is useful to be able to read from a ledger before closing the ledger. To enable such a feature, the writer has to be able to communicate to a reader how many entries it has been able to write successfully. The main idea of this jira is to continuously update a znode with the number of successful writes, and a reader can, for example, watch the node for changes. I was thinking of having a configuration parameter to state how often a writer should update the hint on ZooKeeper (e.g., every 1000 requests, every 10,000 requests). Clearly updating more often increases the overhead of writing to ZooKeeper, although the impact on the performance of writes to BookKeeper should be minimal given that we make an asynchronous call to update the hint. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-462) Last hint for open ledger
[ https://issues.apache.org/jira/browse/ZOOKEEPER-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737609#action_12737609 ] Patrick Hunt commented on ZOOKEEPER-462: Flavio, what do you want this in? 3.3 or does it have to go into the fix release as well? Last hint for open ledger - Key: ZOOKEEPER-462 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-462 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Flavio Paiva Junqueira Attachments: ZOOKEEPER-462.patch In some use cases of BookKeeper, it is useful to be able to read from a ledger before closing the ledger. To enable such a feature, the writer has to be able to communicate to a reader how many entries it has been able to write successfully. The main idea of this jira is to continuously update a znode with the number of successful writes, and a reader can, for example, watch the node for changes. I was thinking of having a configuration parameter to state how often a writer should update the hint on ZooKeeper (e.g., every 1000 requests, every 10,000 requests). Clearly updating more often increases the overhead of writing to ZooKeeper, although the impact on the performance of writes to BookKeeper should be minimal given that we make an asynchronous call to update the hint. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-462) Last hint for open ledger
[ https://issues.apache.org/jira/browse/ZOOKEEPER-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12737791#action_12737791 ] Hadoop QA commented on ZOOKEEPER-462: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12414483/ZOOKEEPER-462.patch against trunk revision 799741. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/164/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/164/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-vesta.apache.org/164/console This message is automatically generated. Last hint for open ledger - Key: ZOOKEEPER-462 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-462 Project: Zookeeper Issue Type: New Feature Components: contrib-bookkeeper Reporter: Flavio Paiva Junqueira Assignee: Flavio Paiva Junqueira Attachments: ZOOKEEPER-462.patch In some use cases of BookKeeper, it is useful to be able to read from a ledger before closing the ledger. To enable such a feature, the writer has to be able to communicate to a reader how many entries it has been able to write successfully. The main idea of this jira is to continuously update a znode with the number of successful writes, and a reader can, for example, watch the node for changes. I was thinking of having a configuration parameter to state how often a writer should update the hint on ZooKeeper (e.g., every 1000 requests, every 10,000 requests). Clearly updating more often increases the overhead of writing to ZooKeeper, although the impact on the performance of writes to BookKeeper should be minimal given that we make an asynchronous call to update the hint. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
