[
https://issues.apache.org/jira/browse/BOOKKEEPER-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16039378#comment-16039378
]
Hudson commented on BOOKKEEPER-944:
-----------------------------------
UNSTABLE: Integrated in Jenkins build bookkeeper-master #1776 (See
[https://builds.apache.org/job/bookkeeper-master/1776/])
BOOKKEEPER-944: LowWaterMark Storage Threshold (sijie: rev
efd8ec26926d2e75f51d6d576543a0dcc116b83f)
* (edit)
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
* (add)
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerDirsMonitor.java
* (edit)
bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/IndexPersistenceMgrTest.java
* (edit)
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerDirsManager.java
* (edit)
bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java
* (edit) bookkeeper-server/conf/bk_server.conf
* (edit)
bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/BookieInitializationTest.java
* (edit)
bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/TestLedgerDirsManager.java
* (edit)
bookkeeper-server/src/main/java/org/apache/bookkeeper/util/DiskChecker.java
> Multiple issues and improvements to BK Compaction.
> --------------------------------------------------
>
> Key: BOOKKEEPER-944
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-944
> Project: Bookkeeper
> Issue Type: Improvement
> Components: bookkeeper-server
> Affects Versions: 4.4.0
> Reporter: Venkateswararao Jujjuri (JV)
> Assignee: Venkateswararao Jujjuri (JV)
> Fix For: 4.5.0
>
>
> We have identified multiple issues with BK compaction.
> This issue is to list all of them in one Jira ticket.
> 1.
> MajorCompaction and MinorCompaction are very basic. Either they do it or
> won’t do it. Proposal is to add Low Water Mark(LWM) and High Water Mark(HWM)
> to the disk space. Have different compaction frequency and re-claim %s when
> the disk space is < low water mark , > LWM < HWM, > HWM.
> 2.
> MajorCompaction and Minor Compactions are strictly frequency based. They
> should at least time of the day based, and also run during low system load,
> and if the system load raises, reduce the compaction depending on the disk
> availability
> 3.
> Current code disables compaction when disk space grows beyond configured
> threshold. There is no exit from this point. Have an option to keep reserved
> space for compaction, at least 2 entryLog file sizes when
> isForceGCAllowWhenNoSpace enabled.
> 4.
> Current code toggles READONLY status of the bookie as soon as it falls below
> the disk storage threshold. Imagine if we keep 95% as the threshold, Bookie
> becomes RW as soon as it falls below 95 % and few more writes pushes it above
> 95 and it turns back to RONLY. Use a set of defines (another set of LWM/HWM?)
> where Bookie turns RO on high end and won't become RW until it hits low end.
> 5.
> Current code never checks if the compaction is enabled or disabled once the
> major/minor compaction is started. If the bookie goes > disk threshold (95%)
> and at that compaction is going on, it never checks until it finishes but
> there may not be disk available for compaction to take place. So check if
> compaction is enabled after processing every EntryLog.
> 6.
> Current code changes the Bookie Cookie value even when new storage is added.
> When the cookie changes Bookie becomes a new one, and BK cluster treats it as
> new bookie. If we have mechanism to keep valid cookie even after adding
> additional disk space, we may have a chance to bring the bookie back to
> healthy mode and have compaction going.
> 7. Bug
> CheckPoint was never attempted to complete after once sync failure. There is
> a TODO in the code for this area.
> 8.
> When the disk is above threshold, Bookie goes to RO. If we have to restart
> the bookie, on the way back, bookie tries to create new entrylog and other
> files, which will fail because disk usage is above threshold, hence bookie
> refuses to come up.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
