[
https://issues.apache.org/jira/browse/BOOKKEEPER-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13758039#comment-13758039
]
Sijie Guo commented on BOOKKEEPER-664:
--------------------------------------
{quote}
EntryLogger#flush() and EntryLogger#addEntry() being synced.
{quote}
the checkpoint mechanism in BOOKKEEPER-564 is trying to resolving this
synchronization by avoiding flushing current entry log.
so similar idea could be applied here, we don't need to flush the entry logger
after compaction. we could wait until it rolls to write new entry logger file
and flushes the previous entry logger files. it is easy to do that using the
EntryLogListener callback.
BTW, I don't think you use max outstanding requests in the right way. you
should not flush entry log when reaching max outstanding requests. you could
just add the entry back to ledger storage and let it go through the original
ledger storage add flow (since we already optimize the flush/checkpoint flow).
the only thing that Compaction Scanner could do is to use
max_outstanding_requests as a rate limiter in compaction scanner to limit the
rate of compaction adds (e.g. 10% of bandwidth used for compaction).
> Compaction increases latency on journal writes
> ----------------------------------------------
>
> Key: BOOKKEEPER-664
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-664
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.1
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.2.2, 4.3.0
>
> Attachments:
> 0001-BOOKKEEPER-664-branch-4.2-Compaction-increases-laten.patch,
> 0001-BOOKKEEPER-664-Compaction-increases-latency-on-journ_branch4.2.patch,
> 0001-BOOKKEEPER-664-Compaction-increases-latency-on-journ-branch-4.2.patch,
> 0001-BOOKKEEPER-664-Compaction-increases-latency-on-journ.patch,
> 0001-BOOKKEEPER-664-Compaction-increases-latency-on-journ_trunk.patch,
> 0002-BOOKKEEPER-664-trunk-Compaction-increases-latency-on.patch, bench.png
>
>
> Compaction writes to the journal to avoid data loss (see BOOKKEEPER-530).
> BOOKKEEPER-530 correctly identified that this may affect latency on the
> journal but we have observed this since in production. It is possible to
> avoid the journal completely, as twitter do in their github branch.
> Basically, we need to write to the entrylogger first, flush the entry log and
> then add to the index.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
