Luo Chen has posted comments on this change. Change subject: Implemented Disk Components Alignment Based on IDs ......................................................................
Patch Set 6: (1 comment) https://asterix-gerrit.ics.uci.edu/#/c/1761/6/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/CorrelatedPrefixMergePolicy.java File asterixdb/asterix-common/src/main/java/org/apache/asterix/common/context/CorrelatedPrefixMergePolicy.java: PS6, Line 103: isMergeLagging( > I think that there is no need for this method after the introduction of com >From my experiments, we would still need this method for flow control. Here is >the reason. Suppose we have a dataset D with a BTree primary index P and InvertedIndex secondary index S. Note that the merge operation of S is much slower than P (one problem for InvertedIndex). Then a lot of data come in. The first time when the merge policy decides to merge P (say merge 5 components), it would also send a merge request on behalf of S. However, the second time when the merge policy decides to merge P, since the merge of S is still ongoing, S would be ignored for this time (there couldn't be two concurrent merge operations for the same index). The same thing could happen for the third time. After a while, when S finishes its merge operation, there could be a lot of disk components accumulated. The next time when the policy decides to merge, it could merge a lot of lagged disk components (from my observation it could be 50 or more). This problem would get even worse and worse, eventually blowing up the system. -- To view, visit https://asterix-gerrit.ics.uci.edu/1761 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I768ee9ac0a8d3c99c631086093a6b778b2e7588e Gerrit-PatchSet: 6 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen <[email protected]> Gerrit-Reviewer: Ian Maxon <[email protected]> Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-Reviewer: Jianfeng Jia <[email protected]> Gerrit-Reviewer: Luo Chen <[email protected]> Gerrit-Reviewer: Till Westmann <[email protected]> Gerrit-Reviewer: Yingyi Bu <[email protected]> Gerrit-Reviewer: abdullah alamoudi <[email protected]> Gerrit-HasComments: Yes
