[
https://issues.apache.org/jira/browse/CARBONDATA-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17221222#comment-17221222
]
Yahui Liu commented on CARBONDATA-4044:
---------------------------------------
I have faced this issue, and try to solve it. Currently we will call
CarbonLoaderUtil.checkAndCreateCarbonDataLocation to check and create
Segment_XXX folder(if not exist), but we didn't check wherther stale data exist
in segment folder when Segment_XXX folder already exists. My idea is to try to
remove Segment_XXX folder always before creating Segment_XXX folder again. It
will make sure there will be no stale data. Is this solution validation for all
cases? Please provide some ideas to me. Thanks.
> Fix dirty data in indexfile while IUD with stale data in segment folder
> -----------------------------------------------------------------------
>
> Key: CARBONDATA-4044
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4044
> Project: CarbonData
> Issue Type: Bug
> Reporter: Xingjun Hao
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> XX.mergecarbonindex and XX..segment records the indexfiles list of a segment.
> now, we generate xx.mergeindexfile and xx.segment based on filter out all
> indexfiles(including carbonindex and mergecarbonindex), which will leading
> dirty data when there is stale data in segment folder.
> For example, there are a stale index file in segment_0 folder,
> "0_1603763776.carbonindex".
> While loading, a new carbonindex "0_16037752342.carbonindex" is wrote, when
> merge carbonindex files, we expect to only merge 0_16037752342.carbonindex,
> But If we filter out all carbonindex in segment folder, both
> "0_1603763776.carbonindex" and 0_16037752342.carbonindex will be merged and
> recorded into segment file.
>
> While updating, there has same problem.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
