[
https://issues.apache.org/jira/browse/BOOKKEEPER-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13759333#comment-13759333
]
Sijie Guo commented on BOOKKEEPER-664:
--------------------------------------
{quote}
The attached patch falls to the same issue that caused us to go through the
journal in the first place. We need to defer the add to the index, otherwise
the index could be flushed before entrylog, and a crash would cause entry loss.
{quote}
yes. the attached patch is just a quick draft to show the idea I described
previous comment.
thanks for merging the throttling and listener into the patch.
several comments on your latest patch:
1) it would be better to add a test case to cover the case you described in the
comment to ensure future changes would not break it.
2) rotate listener defers the flushing, which would break some test cases in
compaction test if I remembered correctly. so you might need to change those
test cases.
3) add compactionMaxOutstandingRequests and compactionRate to document & config
file
4) use {} rather than "+" in log statements.
5) you might need to notify flushLock when shutting down similar as what I do
in the patch. since entry logger might not rotate during shutting down.
> Compaction increases latency on journal writes
> ----------------------------------------------
>
> Key: BOOKKEEPER-664
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-664
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.1
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.2.2, 4.3.0
>
> Attachments:
> 0001-BOOKKEEPER-664-branch-4.2-Compaction-increases-laten.patch,
> 0001-BOOKKEEPER-664-branch-4.2-Compaction-increases-laten.patch,
> 0001-BOOKKEEPER-664-Compaction-increases-latency-on-journ_branch4.2.patch,
> 0001-BOOKKEEPER-664-Compaction-increases-latency-on-journ-branch-4.2.patch,
> 0001-BOOKKEEPER-664-Compaction-increases-latency-on-journ.patch,
> 0001-BOOKKEEPER-664-Compaction-increases-latency-on-journ_trunk.patch,
> 0002-BOOKKEEPER-664-trunk-Compaction-increases-latency-on.patch, bench.png,
> BOOKKEEPER-664.patch
>
>
> Compaction writes to the journal to avoid data loss (see BOOKKEEPER-530).
> BOOKKEEPER-530 correctly identified that this may affect latency on the
> journal but we have observed this since in production. It is possible to
> avoid the journal completely, as twitter do in their github branch.
> Basically, we need to write to the entrylogger first, flush the entry log and
> then add to the index.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
