[
https://issues.apache.org/jira/browse/LUCENE-8048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16247854#comment-16247854
]
Nikolay Martynov commented on LUCENE-8048:
------------------------------------------
Just to clarify:
* On Linux directory {{fsync}} does work, but it doesn't solve the problem
because this {{fsync}} happens after segment list has been created. Essentially
guarantees about fsync (weak as they are) are in case of failure: before fsync
you can see changes in any combination, after fsync you are guaranteed to see
exactly what was written.
* This can potentially affect any FS that uses non trivial storage for
directories (which is pretty much everything these days). Word on the internet
is that btrfs is capable of doing out of order directory writes.
* 'kernel automatically detects rename pattern' - I think this only works on
some FSs (ext4) and only if certain mount options are present (auto_da_alloc).
And I think generally this is about syncing file data with directory, not
syncing directory as a whole on rename.
> Filesystems do not guarantee order of directories updates
> ---------------------------------------------------------
>
> Key: LUCENE-8048
> URL: https://issues.apache.org/jira/browse/LUCENE-8048
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Nikolay Martynov
>
> Currently when index is written to disk the following sequence of events is
> taking place:
> * write segment file
> * sync segment file
> * write segment file
> * sync segment file
> ...
> * write list of segments
> * sync list of segments
> * rename list of segments
> * sync index directory
> This sequence leads to potential window of opportunity for system to crash
> after 'rename list of segments' but before 'sync index directory' and
> depending on exact filesystem implementation this may potentially lead to
> 'list of segments' being visible in directory while some of the segments are
> not.
> Solution to this is to sync index directory after all segments have been
> written. [This
> commit|https://github.com/mar-kolya/lucene-solr/commit/58e05dd1f633ab9b02d9e6374c7fab59689ae71c]
> shows idea implemented. I'm fairly certain that I didn't find all the places
> this may be potentially happening.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
