[
https://issues.apache.org/jira/browse/KYLIN-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhong Yanghong updated KYLIN-1883:
----------------------------------
Description:
When do the cleanup, current strategy is as follows:
1. firstly create an referenceSet
2. then add items not belonging to the referenceSet to the toDeleteSet
3. finally delete those items in the toDeleteSet
Consensus issue will occur since we cannot make sure that all of the items in
toDeleteSet are not referenced in case that referenceSet changes during the
process.
was:When do the cleanup, current strategy, to firstly create an active
referenceSet and then to create the toDeleteSet by checking whether the whole
set, like dict, is in this referenceSet or not, is not a good solution due to
consensus issues. For example, before the first step, SEGMENT_A is deleted and
leave a DICT_A created at the building step. Then the referenceSet will not
include DICT_A. Between the gap of two steps, SEGMENT_B is starting to build.
Since DICT_A still exists, it can still be referenced by SEGMENT_B. After
SEGMENT_B owns the reference to DICT_A, the step two begins. Then DICT_A will
still be included in the toDeleteSet and will be deleted later. Finally
SEGMENT_B only owns a reference with no data.
> Consensus Problem when running the tool, MetadataCleanupJob
> -----------------------------------------------------------
>
> Key: KYLIN-1883
> URL: https://issues.apache.org/jira/browse/KYLIN-1883
> Project: Kylin
> Issue Type: Bug
> Reporter: Zhong Yanghong
> Assignee: Zhong Yanghong
> Attachments:
> better_solution_for_consensus_issue_of_MetadataCleanupJob.patch
>
>
> When do the cleanup, current strategy is as follows:
> 1. firstly create an referenceSet
> 2. then add items not belonging to the referenceSet to the toDeleteSet
> 3. finally delete those items in the toDeleteSet
> Consensus issue will occur since we cannot make sure that all of the items in
> toDeleteSet are not referenced in case that referenceSet changes during the
> process.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
