[
https://issues.apache.org/jira/browse/SOLR-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17506950#comment-17506950
]
Jan Høydahl commented on SOLR-15051:
------------------------------------
Please consider deleting the central git branch
[https://github.com/apache/solr/tree/jira/solr-15051-blob] and continue
development in a private fork.
> Shared storage -- BlobDirectory (de-duping)
> -------------------------------------------
>
> Key: SOLR-15051
> URL: https://issues.apache.org/jira/browse/SOLR-15051
> Project: Solr
> Issue Type: Improvement
> Reporter: David Smiley
> Assignee: David Smiley
> Priority: Major
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> This proposal is a way to accomplish shared storage in SolrCloud with a few
> key characteristics: (A) using a Directory implementation, (B) delegates to a
> backing local file Directory as a kind of read/write cache (C) replicas have
> their own "space", (D) , de-duplication across replicas via reference
> counting, (E) uses ZK but separately from SolrCloud stuff.
> The Directory abstraction is a good one, and helps isolate shared storage
> from the rest of SolrCloud that doesn't care. Using a backing normal file
> Directory is faster for reads and is simpler than Solr's HDFSDirectory's
> BlockCache. Replicas having their own space solves the problem of multiple
> writers (e.g. of the same shard) trying to own and write to the same space,
> and it implies that any of Solr's replica types can be used along with what
> goes along with them like peer-to-peer replication (sometimes faster/cheaper
> than pulling from shared storage). A de-duplication feature solves needless
> duplication of files across replicas and from parent shards (i.e. from shard
> splitting). The de-duplication feature requires a place to cache directory
> listings so that they can be shared across replicas and atomically updated;
> this is handled via ZooKeeper. Finally, some sort of Solr daemon /
> auto-scaling code should be added to implement "autoAddReplicas", especially
> to provide for a scenario where the leader is gone and can't be replicated
> from directly but we can access shared storage.
> For more about shared storage concepts, consider looking at the description
> in SOLR-13101 and the linked Google Doc.
> *[PROPOSAL
> DOC|https://docs.google.com/document/d/1kjQPK80sLiZJyRjek_Edhokfc5q9S3ISvFRM2_YeL8M/edit?usp=sharing]*
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
