[
https://issues.apache.org/jira/browse/OAK-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Mueller updated OAK-9748:
--------------------------------
Description:
Problem
The current way of dealing with download retry is to build a new FlatFileStore
folder each time a retry happens. Merge daemon thread creates a merge folder
under that FlatFileStore folder and dump intermediate merge progress there.
When there's a download retry, the retried process load all the previously
created FlatFileStore into its progress, however it does not take in the
already merged intermediate files.
The unit test does not run into this issue because only a small number of
intermediate batch merged files are generated and batch merge size by default
is 64 files. So intermediate merge does not happen.
Fix Implementation
Adding ability to forcefully stop merge process, and in retry load the previous
merged files into sorted files list to be merged later.
> Parallel Merge does not handle download retry correctly
> -------------------------------------------------------
>
> Key: OAK-9748
> URL: https://issues.apache.org/jira/browse/OAK-9748
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: indexing
> Environment: The current way of dealing with download retry is to
> build a new FlatFileStore folder each time a retry happens. Merge daemon
> thread creates a merge folder under that FlatFileStore folder and dump
> intermediate merge progress there.
> When there's a download retry, the retried process load all the previously
> created FlatFileStore into its progress, however it does not take in the
> already merged intermediate files.
> The unit test does not run into this issue because only a small number of
> intermediate batch merged files are generated and batch merge size by default
> is 64 files. So intermediate merge does not happen.
> Reporter: Thomas Mueller
> Priority: Major
>
> Problem
> The current way of dealing with download retry is to build a new
> FlatFileStore folder each time a retry happens. Merge daemon thread creates a
> merge folder under that FlatFileStore folder and dump intermediate merge
> progress there.
> When there's a download retry, the retried process load all the previously
> created FlatFileStore into its progress, however it does not take in the
> already merged intermediate files.
> The unit test does not run into this issue because only a small number of
> intermediate batch merged files are generated and batch merge size by default
> is 64 files. So intermediate merge does not happen.
> Fix Implementation
> Adding ability to forcefully stop merge process, and in retry load the
> previous merged files into sorted files list to be merged later.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
