[
https://issues.apache.org/jira/browse/BOOKKEEPER-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13642467#comment-13642467
]
Sijie Guo edited comment on BOOKKEEPER-564 at 4/26/13 1:19 AM:
---------------------------------------------------------------
{quote}
Journal implements CheckpointListener. LedgerStorage takes a Checkpoint
listener on construction. SyncThread is broken out from Bookie, and
LedgerStorage constructs and owns it, but the implementation is separate from
any particular LedgerStorage implementation.
{quote}
{quote}
Journal implements CheckpointListener#checkpointOccurred(CheckPoint) which
moves the "lastmark" pointer.
{quote}
it doesn't make any sense to make ledger storage control journal to move the
mark. the checkpoint flow is not simple a responsibility of a single component.
A bookie's data has two parts, one is persisted indexed data, the other is
in-memory indexed data + persisted txns. And a checkpoint is the boundary of
these two parts. The SyncThread is the role to change the data from one state
(in-memory + txns) to the other state (permanent indexable): 1) ledger storage
persist its in-memory data, 2) journal move its pointer and gc journals if
necessary, 3) if any is failed due to disk issue, it means that such kind of
data conversion could not proceed any more, the bookie should be turned to
readonly at this moment (otherwise, it would be OOM). when you moved sync
thread to ledger storage, you are moving the 2) and 3) part implicitly to
ledger storage, which make ledger storage too ambitious and the flow unclear.
was (Author: hustlmsp):
{quote}
Journal implements CheckpointListener. LedgerStorage takes a Checkpoint
listener on construction. SyncThread is broken out from Bookie, and
LedgerStorage constructs and owns it, but the implementation is separate from
any particular LedgerStorage implementation.
{quote}
{quote}
Journal implements CheckpointListener#checkpointOccurred(CheckPoint) which
moves the "lastmark" pointer.
{quote}
it doesn't make any sense to make ledger storage control journal to move the
mark. the checkpoint flow is not simple a responsibility of a single component.
A bookie's data has two parts, one is persisted indexed data, the other is
in-memory indexed data + persisted txns. And a checkpoint is the boundary of
these two parts. The SyncThread is the role to change the data from one state
(in-memory + txns) to the other state (permanent indexable): 1) ledger storage
persist its data, 2) journal move its pointer and gc journals if necessary, 3)
if any is failed due to disk issue, it means that such kind of data conversion
could not proceed any more, the bookie should be turned to readonly at this
moment (otherwise, it would be OOM). when you moved sync thread to ledger
storage, you are moving the 2) and 3) part implicitly to ledger storage, which
make ledger storage too ambitious and the flow unclear.
> Better checkpoint mechanism
> ---------------------------
>
> Key: BOOKKEEPER-564
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-564
> Project: Bookkeeper
> Issue Type: Improvement
> Components: bookkeeper-server
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.3.0
>
> Attachments: 0001-BOOKKEEPER-564-Better-checkpoint-mechanism.patch,
> 0001-BOOKKEEPER-564-Better-checkpoint-mechanism.patch,
> 0002-BOOKKEEPER-564-Better-checkpoint-mechanism.patch, BOOKKEEPER-564.patch,
> BOOKKEEPER-564.patch, BOOKKEEPER-564.patch
>
>
> Currently, SyncThread made a checkpoint too frequently, which affects
> performance. data is writing to entry logger file might be blocked by syncing
> same entry logger file, which affect bookie to achieve higher throughput. We
> could schedule checkpoint only when rotating an entry log file. so new
> incoming entries would be written to newer entry log file and old entry log
> file could be synced.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
