[jira] [Commented] (SOLR-15051) Shared storage -- BlobDirectory (de-duping)
[ https://issues.apache.org/jira/browse/SOLR-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283328#comment-17283328 ] David Smiley commented on SOLR-15051: - Linking SOLR-15089 for the S3 impl of BackupRepository, which is lightly a dependency to BlobDirectory. Some next steps, un-ordered: * Testing of what we have via existing tests. Maybe some sub-set of tests using MiniSolrCloudCluster? * Switch to BackupRepository API for backing storage API * Add first draft "Listings" component implementation with many limitations (trivial in-JVM static global (no ZK), no de-duping) Then: * Listings: ZK * Listings: De-duping > Shared storage -- BlobDirectory (de-duping) > --- > > Key: SOLR-15051 > URL: https://issues.apache.org/jira/browse/SOLR-15051 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: David Smiley >Assignee: David Smiley >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > This proposal is a way to accomplish shared storage in SolrCloud with a few > key characteristics: (A) using a Directory implementation, (B) delegates to a > backing local file Directory as a kind of read/write cache (C) replicas have > their own "space", (D) , de-duplication across replicas via reference > counting, (E) uses ZK but separately from SolrCloud stuff. > The Directory abstraction is a good one, and helps isolate shared storage > from the rest of SolrCloud that doesn't care. Using a backing normal file > Directory is faster for reads and is simpler than Solr's HDFSDirectory's > BlockCache. Replicas having their own space solves the problem of multiple > writers (e.g. of the same shard) trying to own and write to the same space, > and it implies that any of Solr's replica types can be used along with what > goes along with them like peer-to-peer replication (sometimes faster/cheaper > than pulling from shared storage). A de-duplication feature solves needless > duplication of files across replicas and from parent shards (i.e. from shard > splitting). The de-duplication feature requires a place to cache directory > listings so that they can be shared across replicas and atomically updated; > this is handled via ZooKeeper. Finally, some sort of Solr daemon / > auto-scaling code should be added to implement "autoAddReplicas", especially > to provide for a scenario where the leader is gone and can't be replicated > from directly but we can access shared storage. > For more about shared storage concepts, consider looking at the description > in SOLR-13101 and the linked Google Doc. > *[PROPOSAL > DOC|https://docs.google.com/document/d/1kjQPK80sLiZJyRjek_Edhokfc5q9S3ISvFRM2_YeL8M/edit?usp=sharing]* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15051) Shared storage -- BlobDirectory (de-duping)
[ https://issues.apache.org/jira/browse/SOLR-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264214#comment-17264214 ] Jason Gerlowski commented on SOLR-15051: Some overall thoughts on the proposal: # I really like the conceptual lines that this draws. Relying strictly on the Directory/DirectoryFactory interface is promising in terms of keeping storage concerns and SolrCloud concerns separate. We've tried Directory-based abstractions before of course (with HdfsDirectory), but this proposal improves on that in concrete ways: index file deduplication/ref-counting, removal of "BlockCache" concept, etc. (This isn't a knock on HdfsDirectory - TLOG/PULL replica types weren't around when HdfsDirectory was introduced, which is really what makes the BlobDirectory design feasible afaict.) # At the risk of counting unhatched chickens - I also think it's promising that this design can piggy-back on some of the SIP-12 work: especially the concrete BackupRepository implementations that SIP-12 proposes for common blob stores. # One specific worry I have is that BlobDirectory methods might be insufficient for accurately refcounting files. e.g. If a replica is deleted while the hosting Solr node is down, what will delete the corresponding "space" for that replica in the blob store, decrement the refcounts of shared files, etc? The proposal describes this being done by BlobDirectory - but when the hosting node is down seemingly the relevant BlobDirectory won't be instantiated anywhere to perform those actions. That said I don't know enough about how Directory objects are instantiated and used to say whether this is actually a real concern or whether existing SolrCloud logic will handle these cases appropriately. Overall I'm in favor of the proposal here as a more flexible alternative (replacement?) for HdfsDirectory. So, +1. > Shared storage -- BlobDirectory (de-duping) > --- > > Key: SOLR-15051 > URL: https://issues.apache.org/jira/browse/SOLR-15051 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: David Smiley >Assignee: David Smiley >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > This proposal is a way to accomplish shared storage in SolrCloud with a few > key characteristics: (A) using a Directory implementation, (B) delegates to a > backing local file Directory as a kind of read/write cache (C) replicas have > their own "space", (D) , de-duplication across replicas via reference > counting, (E) uses ZK but separately from SolrCloud stuff. > The Directory abstraction is a good one, and helps isolate shared storage > from the rest of SolrCloud that doesn't care. Using a backing normal file > Directory is faster for reads and is simpler than Solr's HDFSDirectory's > BlockCache. Replicas having their own space solves the problem of multiple > writers (e.g. of the same shard) trying to own and write to the same space, > and it implies that any of Solr's replica types can be used along with what > goes along with them like peer-to-peer replication (sometimes faster/cheaper > than pulling from shared storage). A de-duplication feature solves needless > duplication of files across replicas and from parent shards (i.e. from shard > splitting). The de-duplication feature requires a place to cache directory > listings so that they can be shared across replicas and atomically updated; > this is handled via ZooKeeper. Finally, some sort of Solr daemon / > auto-scaling code should be added to implement "autoAddReplicas", especially > to provide for a scenario where the leader is gone and can't be replicated > from directly but we can access shared storage. > For more about shared storage concepts, consider looking at the description > in SOLR-13101 and the linked Google Doc. > *[PROPOSAL > DOC|https://docs.google.com/document/d/1kjQPK80sLiZJyRjek_Edhokfc5q9S3ISvFRM2_YeL8M/edit?usp=sharing]* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15051) Shared storage -- BlobDirectory (de-duping)
[ https://issues.apache.org/jira/browse/SOLR-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258633#comment-17258633 ] Jason Gerlowski commented on SOLR-15051: bq. In Solr we have a "BackupRepository" abstraction ... but it might as well be called something like "RemoteStorageRepository" that is not just usable for Backup/restore but also as a backing abstraction for shared storage generally. That sounds reasonable to me, both in terms of naming and in terms of expanding the intended purpose of that interface. (In fact, this is very related to some of the discussions I'm trying to start in SIP-12, a part of which aims at better support for backup/restore to blob stores. I've been wondering how we might be able to align our efforts. At first glance, generalizing BackupRepository seems like it'd be a good way to push in the same direction without stepping on toes. Though I need to think about it more to be more sure.) > Shared storage -- BlobDirectory (de-duping) > --- > > Key: SOLR-15051 > URL: https://issues.apache.org/jira/browse/SOLR-15051 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: David Smiley >Assignee: David Smiley >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > This proposal is a way to accomplish shared storage in SolrCloud with a few > key characteristics: (A) using a Directory implementation, (B) delegates to a > backing local file Directory as a kind of read/write cache (C) replicas have > their own "space", (D) , de-duplication across replicas via reference > counting, (E) uses ZK but separately from SolrCloud stuff. > The Directory abstraction is a good one, and helps isolate shared storage > from the rest of SolrCloud that doesn't care. Using a backing normal file > Directory is faster for reads and is simpler than Solr's HDFSDirectory's > BlockCache. Replicas having their own space solves the problem of multiple > writers (e.g. of the same shard) trying to own and write to the same space, > and it implies that any of Solr's replica types can be used along with what > goes along with them like peer-to-peer replication (sometimes faster/cheaper > than pulling from shared storage). A de-duplication feature solves needless > duplication of files across replicas and from parent shards (i.e. from shard > splitting). The de-duplication feature requires a place to cache directory > listings so that they can be shared across replicas and atomically updated; > this is handled via ZooKeeper. Finally, some sort of Solr daemon / > auto-scaling code should be added to implement "autoAddReplicas", especially > to provide for a scenario where the leader is gone and can't be replicated > from directly but we can access shared storage. > For more about shared storage concepts, consider looking at the description > in SOLR-13101 and the linked Google Doc. > *[PROPOSAL > DOC|https://docs.google.com/document/d/1kjQPK80sLiZJyRjek_Edhokfc5q9S3ISvFRM2_YeL8M/edit?usp=sharing]* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15051) Shared storage -- BlobDirectory (de-duping)
[
https://issues.apache.org/jira/browse/SOLR-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258466#comment-17258466
]
Kevin Risden commented on SOLR-15051:
-
{quote}"Hadoop filesystem interface" is an ideal choice{quote}
Not sure I would go so far as an "ideal choice" - it definitely brings in a lot
of dependencies sadly. The nice part is that it has implementations for local,
hdfs, s3, adls, and gcp at least. So you get multi cloud for free. What I do
not know is how efficient each of those implementations are - especially the
"local" one for file:// type paths.
Just wanted to point out the Hadoop filesystem connector stuff is separate from
the full running of HDFS. Here are a few references:
* General overview:
http://hadoop.apache.org/docs/current3/hadoop-project-dist/hadoop-common/filesystem/index.html
* AWS:
http://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
* Azure: http://hadoop.apache.org/docs/current3/hadoop-azure/index.html
* Google: https://github.com/GoogleCloudDataproc/hadoop-connectors
*
> Shared storage -- BlobDirectory (de-duping)
> ---
>
> Key: SOLR-15051
> URL: https://issues.apache.org/jira/browse/SOLR-15051
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This proposal is a way to accomplish shared storage in SolrCloud with a few
> key characteristics: (A) using a Directory implementation, (B) delegates to a
> backing local file Directory as a kind of read/write cache (C) replicas have
> their own "space", (D) , de-duplication across replicas via reference
> counting, (E) uses ZK but separately from SolrCloud stuff.
> The Directory abstraction is a good one, and helps isolate shared storage
> from the rest of SolrCloud that doesn't care. Using a backing normal file
> Directory is faster for reads and is simpler than Solr's HDFSDirectory's
> BlockCache. Replicas having their own space solves the problem of multiple
> writers (e.g. of the same shard) trying to own and write to the same space,
> and it implies that any of Solr's replica types can be used along with what
> goes along with them like peer-to-peer replication (sometimes faster/cheaper
> than pulling from shared storage). A de-duplication feature solves needless
> duplication of files across replicas and from parent shards (i.e. from shard
> splitting). The de-duplication feature requires a place to cache directory
> listings so that they can be shared across replicas and atomically updated;
> this is handled via ZooKeeper. Finally, some sort of Solr daemon /
> auto-scaling code should be added to implement "autoAddReplicas", especially
> to provide for a scenario where the leader is gone and can't be replicated
> from directly but we can access shared storage.
> For more about shared storage concepts, consider looking at the description
> in SOLR-13101 and the linked Google Doc.
> *[PROPOSAL
> DOC|https://docs.google.com/document/d/1kjQPK80sLiZJyRjek_Edhokfc5q9S3ISvFRM2_YeL8M/edit?usp=sharing]*
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15051) Shared storage -- BlobDirectory (de-duping)
[
https://issues.apache.org/jira/browse/SOLR-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253791#comment-17253791
]
David Smiley commented on SOLR-15051:
-
In Solr we have a "BackupRepository" abstraction already for backup/restore. I
rather like it; it's somewhat generic with a bias towards working with Lucene.
It has "Backup" in it's name but it might as well be called something like
"RemoteStorageRepository" that is not just usable for Backup/restore but also
as a backing abstraction for shared storage generally. Interestingly, I see it
uses the Lucene {{IndexInput}} for reading (instead of an InputStream), and
supports copying via Lucene's {{Directory}} abstraction as well, CC
[~gerlowskija] [~atris] [~varun] [~hgadre]
> Shared storage -- BlobDirectory (de-duping)
> ---
>
> Key: SOLR-15051
> URL: https://issues.apache.org/jira/browse/SOLR-15051
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This proposal is a way to accomplish shared storage in SolrCloud with a few
> key characteristics: (A) using a Directory implementation, (B) delegates to a
> backing local file Directory as a kind of read/write cache (C) replicas have
> their own "space", (D) , de-duplication across replicas via reference
> counting, (E) uses ZK but separately from SolrCloud stuff.
> The Directory abstraction is a good one, and helps isolate shared storage
> from the rest of SolrCloud that doesn't care. Using a backing normal file
> Directory is faster for reads and is simpler than Solr's HDFSDirectory's
> BlockCache. Replicas having their own space solves the problem of multiple
> writers (e.g. of the same shard) trying to own and write to the same space,
> and it implies that any of Solr's replica types can be used along with what
> goes along with them like peer-to-peer replication (sometimes faster/cheaper
> than pulling from shared storage). A de-duplication feature solves needless
> duplication of files across replicas and from parent shards (i.e. from shard
> splitting). The de-duplication feature requires a place to cache directory
> listings so that they can be shared across replicas and atomically updated;
> this is handled via ZooKeeper. Finally, some sort of Solr daemon /
> auto-scaling code should be added to implement "autoAddReplicas", especially
> to provide for a scenario where the leader is gone and can't be replicated
> from directly but we can access shared storage.
> For more about shared storage concepts, consider looking at the description
> in SOLR-13101 and the linked Google Doc.
> *[PROPOSAL
> DOC|https://docs.google.com/document/d/1kjQPK80sLiZJyRjek_Edhokfc5q9S3ISvFRM2_YeL8M/edit?usp=sharing]*
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15051) Shared storage -- BlobDirectory (de-duping)
[
https://issues.apache.org/jira/browse/SOLR-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253696#comment-17253696
]
David Smiley commented on SOLR-15051:
-
Thanks for the look [~krisden]!
bq. There will be a performance penalty here coming from remote storage.
For read performance, if cached locally (BlobDirectory does this), there isn't.
BlobDirectory simply delegates reads to {{MMapDirectory}}. I've seen
[~tpot]'s presentation showing that {{HdfsDirectory}} has a read-time
performance hit.
bq. The BlockCache is off heap typically with Java direct memory so shouldn't
require a large Java heap.
Oops; thanks for the correction!
RE storage APIs/abstractions: Are you claiming that the "Hadoop filesystem
interface" is an ideal choice for a BlobDirectory backing store abstraction?
BlobDirectory or whatever shared system needs to write to *something*, so I'm
sincere in asking for your opinion on what that something should be. I have
basically no HDFS experience so I was unaware of it's generic API. Even if
that interface is nice... I suspect BlobDirectory ought to have some simple
abstraction any way but I'm really not hung up on this choice. I'm leery of
adding heavy to too many dependencies.
> Shared storage -- BlobDirectory (de-duping)
> ---
>
> Key: SOLR-15051
> URL: https://issues.apache.org/jira/browse/SOLR-15051
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
>
> This proposal is a way to accomplish shared storage in SolrCloud with a few
> key characteristics: (A) using a Directory implementation, (B) delegates to a
> backing local file Directory as a kind of read/write cache (C) replicas have
> their own "space", (D) , de-duplication across replicas via reference
> counting, (E) uses ZK but separately from SolrCloud stuff.
> The Directory abstraction is a good one, and helps isolate shared storage
> from the rest of SolrCloud that doesn't care. Using a backing normal file
> Directory is faster for reads and is simpler than Solr's HDFSDirectory's
> BlockCache. Replicas having their own space solves the problem of multiple
> writers (e.g. of the same shard) trying to own and write to the same space,
> and it implies that any of Solr's replica types can be used along with what
> goes along with them like peer-to-peer replication (sometimes faster/cheaper
> than pulling from shared storage). A de-duplication feature solves needless
> duplication of files across replicas and from parent shards (i.e. from shard
> splitting). The de-duplication feature requires a place to cache directory
> listings so that they can be shared across replicas and atomically updated;
> this is handled via ZooKeeper. Finally, some sort of Solr daemon /
> auto-scaling code should be added to implement "autoAddReplicas", especially
> to provide for a scenario where the leader is gone and can't be replicated
> from directly but we can access shared storage.
> For more about shared storage concepts, consider looking at the description
> in SOLR-13101 and the linked Google Doc.
> *[PROPOSAL
> DOC|https://docs.google.com/document/d/1kjQPK80sLiZJyRjek_Edhokfc5q9S3ISvFRM2_YeL8M/edit?usp=sharing]*
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15051) Shared storage -- BlobDirectory (de-duping)
[
https://issues.apache.org/jira/browse/SOLR-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253596#comment-17253596
]
Kevin Risden commented on SOLR-15051:
-
I can't add a comment to the design doc, but wanted to address potentially
misleading statements around the Solr HDFS integration.
{quote}Has an unfortunate search performance penalty. TODO ___ %. Some
indexing penalty too: ___ %.{quote}
There will be a performance penalty here coming from remote storage. I don't
think this is completely avoidable. The biggest issue is on the indexing side
where we need to ensure that documents are reliably written, but this isn't
exactly fast on remote storage.
{quote}The implementation relies on a “BlockCache”, which means running Solr
with large Java heaps.{quote}
The BlockCache is off heap typically with Java direct memory so shouldn't
require a large Java heap.
{quote}
It’s not a generalized shared storage scheme; it’s HDFS specific. It’s
possible to plug in S3 and Alluxio to this but there is overhead. HDFS is
rather complex to operate, whereas say S3 is provided by cloud hosting
providers natively.
{quote}
I'm not sure I understand this statement. There are a few parts to Hadoop. HDFS
is the storage layer that can be complex to operate. The more interesting part
is the Hadoop filesystem interface that is a semi generic adapter between the
HDFS API and other storage backends (S3, ABFS, GCS, etc). The two pieces are
separate and don't require each other to operate. The Hadoop filesystem
interface provides the abstraction necessary to go between local filesystem to
a lot of other cloud provider storage mechanisms.
There may be some overhead there, but I know there has been a lot of work in
the past 1-2 years where the performance has been improved since there has been
a push for cloud storage support.
> Shared storage -- BlobDirectory (de-duping)
> ---
>
> Key: SOLR-15051
> URL: https://issues.apache.org/jira/browse/SOLR-15051
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
>
> This proposal is a way to accomplish shared storage in SolrCloud with a few
> key characteristics: (A) using a Directory implementation, (B) delegates to a
> backing local file Directory as a kind of read/write cache (C) replicas have
> their own "space", (D) , de-duplication across replicas via reference
> counting, (E) uses ZK but separately from SolrCloud stuff.
> The Directory abstraction is a good one, and helps isolate shared storage
> from the rest of SolrCloud that doesn't care. Using a backing normal file
> Directory is faster for reads and is simpler than Solr's HDFSDirectory's
> BlockCache. Replicas having their own space solves the problem of multiple
> writers (e.g. of the same shard) trying to own and write to the same space,
> and it implies that any of Solr's replica types can be used along with what
> goes along with them like peer-to-peer replication (sometimes faster/cheaper
> than pulling from shared storage). A de-duplication feature solves needless
> duplication of files across replicas and from parent shards (i.e. from shard
> splitting). The de-duplication feature requires a place to cache directory
> listings so that they can be shared across replicas and atomically updated;
> this is handled via ZooKeeper. Finally, some sort of Solr daemon /
> auto-scaling code should be added to implement "autoAddReplicas", especially
> to provide for a scenario where the leader is gone and can't be replicated
> from directly but we can access shared storage.
> For more about shared storage concepts, consider looking at the description
> in SOLR-13101 and the linked Google Doc.
> *[PROPOSAL
> DOC|https://docs.google.com/document/d/1kjQPK80sLiZJyRjek_Edhokfc5q9S3ISvFRM2_YeL8M/edit?usp=sharing]*
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15051) Shared storage -- BlobDirectory (de-duping)
[ https://issues.apache.org/jira/browse/SOLR-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249961#comment-17249961 ] David Smiley commented on SOLR-15051: - BTW I started with a SIP but then I doubted this because BlobDirectory fits an existing abstraction well, without requiring changes elsewhere. Regardless, IMO the point of a SIP is largely to draw attention to important things. At the conclusion of an internal "hack day" I'm doing now with [~broustant] and [~nazerke] to make this thing real, I'll share this more (e.g. the dev list). > Shared storage -- BlobDirectory (de-duping) > --- > > Key: SOLR-15051 > URL: https://issues.apache.org/jira/browse/SOLR-15051 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: David Smiley >Assignee: David Smiley >Priority: Major > > This proposal is a way to accomplish shared storage in SolrCloud with a few > key characteristics: (A) using a Directory implementation, (B) delegates to a > backing local file Directory as a kind of read/write cache (C) replicas have > their own "space", (D) , de-duplication across replicas via reference > counting, (E) uses ZK but separately from SolrCloud stuff. > The Directory abstraction is a good one, and helps isolate shared storage > from the rest of SolrCloud that doesn't care. Using a backing normal file > Directory is faster for reads and is simpler than Solr's HDFSDirectory's > BlockCache. Replicas having their own space solves the problem of multiple > writers (e.g. of the same shard) trying to own and write to the same space, > and it implies that any of Solr's replica types can be used along with what > goes along with them like peer-to-peer replication (sometimes faster/cheaper > than pulling from shared storage). A de-duplication feature solves needless > duplication of files across replicas and from parent shards (i.e. from shard > splitting). The de-duplication feature requires a place to cache directory > listings so that they can be shared across replicas and atomically updated; > this is handled via ZooKeeper. Finally, some sort of Solr daemon / > auto-scaling code should be added to implement "autoAddReplicas", especially > to provide for a scenario where the leader is gone and can't be replicated > from directly but we can access shared storage. > For more about shared storage concepts, consider looking at the description > in SOLR-13101 and the linked Google Doc. > *[PROPOSAL > DOC|https://docs.google.com/document/d/1kjQPK80sLiZJyRjek_Edhokfc5q9S3ISvFRM2_YeL8M/edit?usp=sharing]* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
