[
https://issues.apache.org/jira/browse/BOOKKEEPER-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13757659#comment-13757659
]
Ivan Kelly commented on BOOKKEEPER-664:
---------------------------------------
Benching shows this doesn't quite work :/ I've attached the graph. The segments
to pay attention to are 4:40-5:00 & 5:20-end. The rest is loading data in, so
the throttle is high. I loaded 50 ledgers with enough data to roll into a new
logfile. Then I ran a load on 1 ledger, at throttle 1000. After a couple of
minutes I deleted enough ledgers to trigger compaction. 4:40 is without the
patch, 5:20 is with.
Since we dont write to the journal, we add pressure on the ledger disk, as the
offsets are updated in the index. The ledger disk is in the critical path (see
BOOKKEEPER-572) so increasing the load on it increases the time it will take to
respond to the client. I don't have stats for it, but I suspect that flushing
the index it causing the index page being written to by the non-compaction run,
to be swapped out regularly during compaction. I'm going to run again with a
profiler to see more.
> Compaction increases latency on journal writes
> ----------------------------------------------
>
> Key: BOOKKEEPER-664
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-664
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.1
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.2.2, 4.3.0
>
> Attachments:
> 0001-BOOKKEEPER-664-branch-4.2-Compaction-increases-laten.patch,
> 0001-BOOKKEEPER-664-Compaction-increases-latency-on-journ_branch4.2.patch,
> 0001-BOOKKEEPER-664-Compaction-increases-latency-on-journ-branch-4.2.patch,
> 0001-BOOKKEEPER-664-Compaction-increases-latency-on-journ.patch,
> 0001-BOOKKEEPER-664-Compaction-increases-latency-on-journ_trunk.patch,
> 0002-BOOKKEEPER-664-trunk-Compaction-increases-latency-on.patch, bench.png
>
>
> Compaction writes to the journal to avoid data loss (see BOOKKEEPER-530).
> BOOKKEEPER-530 correctly identified that this may affect latency on the
> journal but we have observed this since in production. It is possible to
> avoid the journal completely, as twitter do in their github branch.
> Basically, we need to write to the entrylogger first, flush the entry log and
> then add to the index.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
