Luo Chen has uploaded a new patch set (#2). Change subject: Avoid always merging old components in prefix policy ......................................................................
Avoid always merging old components in prefix policy Current, the prefix policy always looks at the components from oldest to newest to schedule merge. One negative consequence is that the oldest (largest) component gets merged over and over again until it reaches the size limit. This is undesirable since it takes O(n^2) disk IOs (n is the number of flushed components) to produce a final component. This patch is a temporary fix of this behavior, taken from the idea of HBase compaction policy (https://www.ngdata.com/visualizing-hbase -flushes-and-compactions/). The basic idea is that it introduces some size factor (for now it's 1.2) to control the merge behavior. When the prefix policy finds a sequence of components to merge, we also check the oldest (largest) component in the sequence should be smaller than 1.2*the total size of all younger components. By doing so, we can avoid merging oldest components over and over again, making the disk IOs O(nlog n). Change-Id: I464da3fed38cded0aee7b319a35664eae069a2ba --- M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/PrefixMergePolicy.java M hyracks-fullstack/hyracks/hyracks-tests/hyracks-storage-am-lsm-common-test/pom.xml A hyracks-fullstack/hyracks/hyracks-tests/hyracks-storage-am-lsm-common-test/src/test/java/org/apache/hyracks/storage/am/lsm/common/test/PrefixMergePolicyTest.java 3 files changed, 247 insertions(+), 37 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/18/1818/2 -- To view, visit https://asterix-gerrit.ics.uci.edu/1818 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I464da3fed38cded0aee7b319a35664eae069a2ba Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen <[email protected]> Gerrit-Reviewer: Jenkins <[email protected]>
