[
https://issues.apache.org/jira/browse/BOOKKEEPER-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13637079#comment-13637079
]
Sijie Guo commented on BOOKKEEPER-564:
--------------------------------------
{quote}
We're going to be introducing a new ledger storage to trunk hopefully soon.
This is going to need to be tested and benchmarked extensively. If the
behaviour of the ledger storage is dependendent on the behaviour of the sync
thread, which it is, then it's going to make this job much more awkward. We
will have to reimplement the sync thread.
{quote}
I don't see how we need to reimplement the sync thread for the new ledger
storage. we used the same sync thread for both interleaved and skiplist ledger
storage.
{quote}
This doesn't require the checkpoint interface. It can be achieved by simply
exposing #flush(). I don't think we should do this until there's a strong
requirement for it though.
{quote}
The ticket created here is for such requirement (if you read the description of
this ticket), since we found that we could not get a sustain high throughput on
heavy workload due to blind syncing (in a long run on heavy workload, we
observed zero throughput at some points, which is totally bad for latency
sensitive case. because the ledger storage is hanging on syncing too much data
(old data which need to be synced and newly added data) due to blind syncing).
As I said in last comment, you could use the checkpointer interface to do
periodical sync as before or based on the calculation of number of bytes
unflushed (if you don't like the way let ledger storage offering the hint). but
this interface allowed us and it already helped us to use such hint to optimize
the bookie to get a sustained higher throughput.
> Better checkpoint mechanism
> ---------------------------
>
> Key: BOOKKEEPER-564
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-564
> Project: Bookkeeper
> Issue Type: Improvement
> Components: bookkeeper-server
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.3.0
>
> Attachments: 0001-BOOKKEEPER-564-Better-checkpoint-mechanism.patch,
> 0001-BOOKKEEPER-564-Better-checkpoint-mechanism.patch,
> 0002-BOOKKEEPER-564-Better-checkpoint-mechanism.patch, BOOKKEEPER-564.patch,
> BOOKKEEPER-564.patch
>
>
> Currently, SyncThread made a checkpoint too frequently, which affects
> performance. data is writing to entry logger file might be blocked by syncing
> same entry logger file, which affect bookie to achieve higher throughput. We
> could schedule checkpoint only when rotating an entry log file. so new
> incoming entries would be written to newer entry log file and old entry log
> file could be synced.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
