Yonik Seeley created SOLR-13102:
-----------------------------------
Summary: Shared storage Directory implementation
Key: SOLR-13102
URL: https://issues.apache.org/jira/browse/SOLR-13102
Project: Solr
Issue Type: New Feature
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Yonik Seeley
We need a general strategy (and probably a general base class) that can work
with shared storage and not corrupt indexes from multiple writers.
One strategy that is used on local disk is to use locks. This doesn't extend
well to remote / shared filesystems when the locking is not tied into the
object store itself since a process can lose the lock (a long GC or whatever)
and then immediately try to write a file and there is no way to stop it.
An alternate strategy ditches the use of locks and simply avoids overwriting
files by some algorithmic mechanism.
One of my colleagues outlined one way to do this:
https://www.youtube.com/watch?v=UeTFpNeJ1Fo
That strategy uses random looking filenames and then writes a "core.metadata"
file that maps between the random names and the original names. The problem is
then reduced to overwriting "core.metadata" when you lose the lock. One way to
fix this is to version "core.metadata". Since the new leader election code was
implemented, each shard as a monotonically increasing "leader term", and we can
use that as part of the filename. When a reader goes to open an index, it can
use the latest file from the directory listing, or even use the term obtained
from ZK if we can't trust the directory listing to be up to date.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
