I am also confused on this. Is adding replicas going to increase search
performance? I'm not sure I see the point of any replicas when using
HDFS. Is there one?
Thank you!
-Joe
On 2/25/2015 10:57 AM, Erick Erickson wrote:
bq: And the data sync between leader/replica is always a problem
Not quite sure what you mean by this. There shouldn't need to be
any synching in the sense that the index gets replicated, the
incoming documents should be sent to each node (and indexed
to HDFS) as they come in.
bq: There is duplicate index computing on Replilca side.
Yes, that's the design of SolrCloud, explicitly to provide data safety.
If you instead rely on the leader to index and somehow pull that
indexed form to the replica, then you will lose data if the leader
goes down before sending the indexed form.
bq: My thought is that the leader and the replica all bind to the same data
index directory.
This is unsafe. They would both then try to _write_ to the same
index, which can easily corrupt indexes and/or all but the first
one to access the index would be locked out.
All that said, the HDFS triple-redundancy compounded with the
Solr leaders/replicas redundancy means a bunch of extra
storage. You can turn the HDFS replication down to 1, but that has
other implications.
Best,
Erick
On Tue, Feb 24, 2015 at 11:12 PM, longsan <longsan...@sina.com> wrote:
We used HDFS as our Solr index storage and we really have a heavy update
load. We had met much problems with current leader/replica solution. There
is duplicate index computing on Replilca side. And the data sync between
leader/replica is always a problem.
As HDFS already provides data replication on data layer, could Solr provide
just service layer replication?
My thought is that the leader and the replica all bind to the same data
index directory. And the leader will build up index for new request, the
replica will just keep update the index version with the leader(such as a
soft commit periodically? ). If the leader lost then the replica will take
the duty immediately.
Thanks for any suggestion of this idea.
--
View this message in context:
http://lucene.472066.n3.nabble.com/New-leader-replica-solution-for-HDFS-tp4188735.html
Sent from the Solr - User mailing list archive at Nabble.com.
