This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git
The following commit(s) were added to refs/heads/master by this push:
new c4af734 BP-32: Advisory (optimistic) write close
c4af734 is described below
commit c4af73427378312d9e7d50cc06a9523a78ecaf41
Author: cguttapalem <[email protected]>
AuthorDate: Wed May 2 20:59:04 2018 -0700
BP-32: Advisory (optimistic) write close
Descriptions of the changes in this PR:
initial proposal for Advisory (optimisitic) write close
Master Issue: #1343
Author: cguttapalem <[email protected]>
Reviewers: Enrico Olivelli <[email protected]>, Sijie Guo
<[email protected]>
This closes #1374 from reddycharan/advwriteclosebp
---
site/bps/BP-32-advisory-write-close.md | 54 ++++++++++++++++++++++++++++++++++
site/community/bookkeeper_proposals.md | 5 ++--
2 files changed, 57 insertions(+), 2 deletions(-)
diff --git a/site/bps/BP-32-advisory-write-close.md
b/site/bps/BP-32-advisory-write-close.md
new file mode 100644
index 0000000..f3d0ffc
--- /dev/null
+++ b/site/bps/BP-32-advisory-write-close.md
@@ -0,0 +1,54 @@
+---
+title: "BP-32: Advisory (optimistic) write close"
+issue: https://github.com/apache/bookkeeper/issues/1343
+state: Under Discussion
+release: N/A
+---
+
+### Motivation
+
+With entrylog per ledger feature
(https://github.com/apache/bookkeeper/issues/570) there will be dedicated
entrylog for each ledger and it provides EntryLogManagerForEntryLogPerLedger
(EntryLogManager). Since there is going to be entrylog per ledger, with the
current Bookie implementation there is no way for
EntryLogManagerForEntryLogPerLedger (EntryLogManager) to know when the entrylog
is writeclosed, so that entrylog for the active ledger can be rotated. So it
would be ideal to have ex [...]
+
+### Proposed Changes
+
+So in entrylog per ledger feature implementation, as a first step of handling
this, expireAfterAccess (each entrylog should be automatically rotated from the
current active entrylog list once a fixed duration has elapsed after the last
access of entrylog for addEntry) will be implemented and also the maximum
number of entrylogs that can be active at a given time (this is for limiting
number of entrylogs/filedescriptors open at a given time).
+
+The above mentioned approaches are preliminary ways of handling but it would
be ideal to have explicit call to EntryLogger when the write for this ledger is
done so that it can rotate the entrylog as soon as it is write closed. So to
implement this following can be done
+
+- have explicit write close request. This write close request should be sent
to the current ensemble when the write handle is closed. This should be just
optimistic write close operation and callback of this operation should be just
logger, saying if it is succeeded / partially succeeded / failed. This should
be done asynchronously and the write handle close operation should not be
blocked for this response.
+
+- In the case of ledger recover open, readrequest (ReadEntryProcessorV3) with
fence flag, can take care of calling appropriate methods in ledgerdescriptor /
entry logger to rotate the entrylog for the ledger
+
+- in the case of auto-replication case, LedgerFragmentReplicator already uses
bookieclient for addEntry (bkc.getBookieClient().addEntry), the same
bookieClient instance can be used to call explicit writeClose method in
bookieclient.
+
+- in the case of any write failure in bookie for an entry, then before sending
error response to the client do entrylog rotation
+
+EntryLogManagerForEntryLogPerLedger is the class which is last layer in the
stack to receive this writeClose call for a ledger, it would store this info in
in-memory data structure and when next next checkpoint is called it would
rotate entrylogs of these writeclosed ledgers.
+
+### Public Interfaces
+
+As explained above, this advisory write close should be transparent to
Bookkeeper API user, this should be piggybagged in writehandle close call and
other internal Bookie / BookieClient internal class methods
(ReadEntryProcessorV3.readrequest and LedgerFragmentReplicator). But this
feature introduces new protobuf message between Client and Bookie.
+
+message WriteCloseRequest {
+ required int64 ledgerId = 1;
+ required bytes masterKey = 2;
+}
+
+message WriteCloseResponse {
+ required StatusCode status = 1;
+ required int64 ledgerId = 2;
+}
+
+### Compatibility, Deprecation, and Migration Plan
+
+- With this feature we are introducing new protocol message. Will do the
required Compatibility testing. But since it is going to be advisory
(optimistic) in nature, failure in this request in anyway should not affect
functioning in any way.
+- Also, possibly this explicit write close can be used in future for other
purposes.
+
+### Test Plan
+
+- unit tests for all the write/read handle close scenarios, recoveropen and
replicator scenarios
+- end-to-end integrations tests.
+
+### Rejected Alternatives
+
+N/A
diff --git a/site/community/bookkeeper_proposals.md
b/site/community/bookkeeper_proposals.md
index 9b04499..b9f4396 100644
--- a/site/community/bookkeeper_proposals.md
+++ b/site/community/bookkeeper_proposals.md
@@ -56,7 +56,7 @@ using Google Doc.
- state: "Under Discussion"
- release: leave the release to `N/A`. you can only mark a release
after a BP is implemented.
- Google Doc
- - Make a copy of the
[BP-Template](https://docs.google.com/document/d/1DsmH54LoohgwqnEjESPQNtIYxxcOy2rwonZ_TJCwws0).
Name the BP file as `BP-<number>-[caption-of-proposal]`.
+ - Make a copy of the
[BP-Template](https://docs.google.com/document/d/1DsmH54LoohgwqnEjESPQNtIYxxcOy2rwonZ_TJCwws0).
Name the BP file as `BP-<number>-[caption-of-proposal]`.
- Fill the sections listed in the BP template.
1. Send a PR for this BP. Following the instructions in the pull request
template.
- add `BP` label to this PR
@@ -85,7 +85,7 @@ using Google Doc.
This section lists all the _bookkeeper proposals_ made to BookKeeper.
-*Next Proposal Number: 32*
+*Next Proposal Number: 33*
### Inprogress
@@ -103,6 +103,7 @@ Proposal | State
[BP-29: Metadata API module](../../bps/BP-29-metadata-store-api-module) |
Accepted
[BP-30: BookKeeper Table
Service](https://docs.google.com/document/d/155xAwWv5IdOitHh1NVMEwCMGgB28M3FyMiQSxEpjE-Y/edit#heading=h.56rbh52koe3f)
| Accepted
[BP-31: BookKeeper Durability Anchor](../../bps/BP-31-durability) | Accepted
+[BP-32: Advisory (optimistic) write
close](../../bps/BP-32-advisory-write-close) | Draft
### Adopted
--
To stop receiving notification emails like this one, please contact
[email protected].
