Re: Spread SolrCloud across two locations

[Jan Høydahl]

Thanks for checking Shawn.

So rolling ZK restart is bad, and ZK nodes with different config is bad,
Guess this could still work if
* All ZK config changes are done by stopping ALL zk nodes
* All config changes are done controlled and manual so DC1 don’t come up by 
surprise with old config

PS: I was not proposing an *automatic* triggering of a reconfiguration script, 
but rather to have a script that someone runs manually in order to make sure 
one does not mess up the reconfiguration

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 2. jun. 2017 kl. 14.57 skrev Shawn Heisey :
> 
> On 5/29/2017 8:57 AM, Jan Høydahl wrote:
>> And if you start all three in DC1, you have 3+3 voting, what would
>> then happen? Any chance of state corruption?
>> 
>> I believe that my solution isolates manual change to two ZK nodes in
>> DC2, while your requires config change to 1 in DC2 and manual
>> start/stop of 1 in DC1.
> 
> I took the scenario to the zookeeper user list.  Here's the thread:
> 
> http://zookeeper-user.578899.n2.nabble.com/Yet-another-quot-two-datacenter-quot-discussion-td7583106.html
> 
> I'm not completely clear on what they're saying, but here's what I think
> it means:  Dealing with a loss of dc1 by reconfiguring ZK servers in DC2
> might work, or it might crash and burn once connectivity to DC1 is restored.
> 
>> Well, that’s not up to me to decide, it’s the customer environment
>> that sets the constraints, they currently have 2 independent geo
>> locations. And Solr is just a dependency of some other app they need
>> to install, so doubt that they are very happy to start adding racks or
>> independent power/network for this alone. Of course, if they already
>> have such redundancy within one of the DCs, placing a 3rd ZK there is
>> an ideal solution with probably good enough HA. If not, I’m looking
>> for the 2nd best low-friction approach with software-only.
> 
> Even if all goes well with scripted reconfiguration of DC2, I don't
> think I'd want to try and automate it, because of the chance for a brief
> outage to trigger it.  Without automation, if the failure happened at
> just the wrong moment, it could be a while before anyone notices, and it
> might be hours after it gets noticed before relevant personnel are in a
> position to run the reconfiguration script on DC2, during which you'd
> have a read-only SolrCloud.
> 
> Frequently search is such a critical part of of a web applications that
> if it doesn't work, there IS no web application.  That certainly
> describes the systems that use the Solr installations that I manage. 
> For that kind of application, damage to reputation caused by a couple of
> hours where the website doesn't get any updates might be MUCH more
> expensive than the monthly cost for a virtual private server from a
> hosting company.
> 
> Thanks,
> Shawn
> 

Re: Spread SolrCloud across two locations

[Shawn Heisey]

On 5/29/2017 8:57 AM, Jan Høydahl wrote:
> And if you start all three in DC1, you have 3+3 voting, what would
> then happen? Any chance of state corruption?
>
> I believe that my solution isolates manual change to two ZK nodes in
> DC2, while your requires config change to 1 in DC2 and manual
> start/stop of 1 in DC1.

I took the scenario to the zookeeper user list.  Here's the thread:

http://zookeeper-user.578899.n2.nabble.com/Yet-another-quot-two-datacenter-quot-discussion-td7583106.html

I'm not completely clear on what they're saying, but here's what I think
it means:  Dealing with a loss of dc1 by reconfiguring ZK servers in DC2
might work, or it might crash and burn once connectivity to DC1 is restored.

> Well, that’s not up to me to decide, it’s the customer environment
> that sets the constraints, they currently have 2 independent geo
> locations. And Solr is just a dependency of some other app they need
> to install, so doubt that they are very happy to start adding racks or
> independent power/network for this alone. Of course, if they already
> have such redundancy within one of the DCs, placing a 3rd ZK there is
> an ideal solution with probably good enough HA. If not, I’m looking
> for the 2nd best low-friction approach with software-only.

Even if all goes well with scripted reconfiguration of DC2, I don't
think I'd want to try and automate it, because of the chance for a brief
outage to trigger it.  Without automation, if the failure happened at
just the wrong moment, it could be a while before anyone notices, and it
might be hours after it gets noticed before relevant personnel are in a
position to run the reconfiguration script on DC2, during which you'd
have a read-only SolrCloud.

Frequently search is such a critical part of of a web applications that
if it doesn't work, there IS no web application.  That certainly
describes the systems that use the Solr installations that I manage. 
For that kind of application, damage to reputation caused by a couple of
hours where the website doesn't get any updates might be MUCH more
expensive than the monthly cost for a virtual private server from a
hosting company.

Thanks,
Shawn

Re: Spread SolrCloud across two locations

[Jan Høydahl]

> I believe that my solution isolates manual change to two ZK nodes in DC2, 
> while your requires config change to 1 in DC2 and manual start/stop of 1 in 
> DC1.

Answering my own statement here. Turns out that in order to flip the observer 
bit for one ZK node, you need to touch the config of all the others, quote from 
https://zookeeper.apache.org/doc/trunk/zookeeperObservers.html 
: "Secondly, in 
every server config file, you must add :observer to the server definition line 
of each Observer”. 

So basically you would do some sed magic or keep two versions of configs around 
and switch between them to toggle. Wonder what happens once you have modified 
the config of one ZK but not the rest, will they be confused and start printing 
warnings to the logs? Is it safe to to the observer flip as a rolling update or 
should you take down the whole ZK ensemble for it?

And if you lose DC1, then modify the remaining ZKs in DC2 to form its own 
quorum, then when the DC1 ZKs come up again their config is not aware that some 
nodes in DC2 have changed from observer to follower - is there a risk that DC1 
starts operating alone once it comes up again, or will it try repeatedly to 
contact the Zk nodes in DC2 and then discover that they disagree about the 
voting authority of some nodes or what?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

Re: Spread SolrCloud across two locations

[Jan Høydahl]

> In my setup once DC1 comes back up make sure you start only two nodes.

And if you start all three in DC1, you have 3+3 voting, what would then happen? 
Any chance of state corruption?
I believe that my solution isolates manual change to two ZK nodes in DC2, while 
your requires config change to 1 in DC2 and manual start/stop of 1 in DC1.

> Add another server in either DC1 or DC2, in a separate rack, with independent 
> power etc. As Shawn says below, install the third ZK there. You would satisfy 
> most of your requirements that way.


Well, that’s not up to me to decide, it’s the customer environment that sets 
the constraints, they currently have 2 independent geo locations. And Solr is 
just a dependency of some other app they need to install, so doubt that they 
are very happy to start adding racks or independent power/network for this 
alone. Of course, if they already have such redundancy within one of the DCs, 
placing a 3rd ZK there is an ideal solution with probably good enough HA. If 
not, I’m looking for the 2nd best low-friction approach with software-only.

Thanks for the input all!

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 26. mai 2017 kl. 21.27 skrev Pushkar Raste :
> 
> In my setup once DC1 comes back up make sure you start only two nodes.
> 
> Now bring down the original observer and make it observer again.
> 
> Bring back the third node
> 
> 
> 
> It seems like lot of work compared to Jan's setup but you get 5 voting
> members instead of 3 in normal situation.
> 
> On May 26, 2017 10:56 AM, "Susheel Kumar"  wrote:
> 
>> Thanks, Pushkar, Make sense.  Trying to understand the difference between
>> your setup vs Jan's proposed setup.
>> 
>> - Seems like when DC1 goes down, in your setup we have to bounce *one* from
>> observer to non-observer while in Jan's setup *two* observers to
>> non-observers.  Anything else I am missing
>> 
>> - When DC1 comes back -  with your setup we need to bounce the one
>> non-observer to observer to have 5 nodes quorum otherwise there are 3 + 3
>> observers while with Jan's setup if we don't take any action when DC1 comes
>> back, we are still operational with 5 nodes quorum.  Isn't it?  Or I am
>> missing something.
>> 
>> 
>> 
>> On Fri, May 26, 2017 at 10:07 AM, Pushkar Raste 
>> wrote:
>> 
>>> Damn,
>>> Math is hard
>>> 
>>> DC1 : 3 non observers
>>> DC2 : 2 non observers
>>> 
>>> 3 + 2 = 5 non observers
>>> 
>>> Observers don't participate in voting = non observers participate in
>> voting
>>> 
>>> 5 non observers = 5 votes
>>> 
>>> In addition to the 2 non observer, DC2 also has an observer, which as you
>>> pointed out does not participate in the voting.
>>> 
>>> We still have 5 voting nodes.
>>> 
>>> 
>>> Think of the observer as a standby name node in Hadoop 1.x, where some
>>> intervention needed if the primary name node went down.
>>> 
>>> 
>>> I hope my math makes sense
>>> 
>>> On May 26, 2017 9:04 AM, "Susheel Kumar"  wrote:
>>> 
>>> From ZK documentation, observers do not participate in vote,  so Pushkar,
>>> when you said 5 nodes participate in voting, what exactly you mean?
>>> 
>>> -- Observers are non-voting members of an ensemble which only hear the
>>> results of votes, not the agreement protocol that leads up to them.
>>> 
>>> Per ZK documentation, 3.4 includes observers,  does that mean Jan thought
>>> experiment is practically possible, correct?
>>> 
>>> 
>>> On Fri, May 26, 2017 at 3:53 AM, Rick Leir  wrote:
>>> 
 Jan, Shawn, Susheel
 
 First steps first. First, let's do a fault-tolerant cluster, then
>> maybe a
 _geographically_ fault-tolerant cluster.
 
 Add another server in either DC1 or DC2, in a separate rack, with
 independent power etc. As Shawn says below, install the third ZK there.
>>> You
 would satisfy most of your requirements that way.
 
 cheers -- Rick
 
 
 On 2017-05-23 12:56 PM, Shawn Heisey wrote:
 
> On 5/23/2017 10:12 AM, Susheel Kumar wrote:
> 
>> Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster
>>> in
>> one of lower env with 6 shards/replica in dc1 & 6 shard/replica in
>> dc2
>> (each shard replicated cross data center) with 3 ZK in dc1 and 2 ZK
>> in
>>> dc2.
>> (I didn't have the availability of 3rd data center for ZK so went
>> with
>>> only
>> 2 data center with above configuration) and so far no issues. Its
>> been
>> running fine, indexing, replicating data, serving queries etc. So in
>> my
>> test, setting up single cluster across two zones/data center works
>>> without
>> any issue when there is no or very minimal latency (in my case around
>>> 30ms
>> one way
>> 
> 
> With that setup, if dc2 goes down, you're all good, but if dc1 goes
>>> down,
> you're not.
> 
> There aren't enough ZK servers in dc2 to maintain quorum when dc1 is
> unreachable, and SolrCloud is going to go read-only.  Queries would
>> most
> like

Re: Spread SolrCloud across two locations

[Pushkar Raste]

In my setup once DC1 comes back up make sure you start only two nodes.

 Now bring down the original observer and make it observer again.

Bring back the third node



It seems like lot of work compared to Jan's setup but you get 5 voting
members instead of 3 in normal situation.

On May 26, 2017 10:56 AM, "Susheel Kumar"  wrote:

> Thanks, Pushkar, Make sense.  Trying to understand the difference between
> your setup vs Jan's proposed setup.
>
> - Seems like when DC1 goes down, in your setup we have to bounce *one* from
> observer to non-observer while in Jan's setup *two* observers to
> non-observers.  Anything else I am missing
>
> - When DC1 comes back -  with your setup we need to bounce the one
> non-observer to observer to have 5 nodes quorum otherwise there are 3 + 3
> observers while with Jan's setup if we don't take any action when DC1 comes
> back, we are still operational with 5 nodes quorum.  Isn't it?  Or I am
> missing something.
>
>
>
> On Fri, May 26, 2017 at 10:07 AM, Pushkar Raste 
> wrote:
>
> > Damn,
> > Math is hard
> >
> > DC1 : 3 non observers
> > DC2 : 2 non observers
> >
> > 3 + 2 = 5 non observers
> >
> > Observers don't participate in voting = non observers participate in
> voting
> >
> > 5 non observers = 5 votes
> >
> > In addition to the 2 non observer, DC2 also has an observer, which as you
> > pointed out does not participate in the voting.
> >
> > We still have 5 voting nodes.
> >
> >
> > Think of the observer as a standby name node in Hadoop 1.x, where some
> > intervention needed if the primary name node went down.
> >
> >
> > I hope my math makes sense
> >
> > On May 26, 2017 9:04 AM, "Susheel Kumar"  wrote:
> >
> > From ZK documentation, observers do not participate in vote,  so Pushkar,
> > when you said 5 nodes participate in voting, what exactly you mean?
> >
> > -- Observers are non-voting members of an ensemble which only hear the
> > results of votes, not the agreement protocol that leads up to them.
> >
> > Per ZK documentation, 3.4 includes observers,  does that mean Jan thought
> > experiment is practically possible, correct?
> >
> >
> > On Fri, May 26, 2017 at 3:53 AM, Rick Leir  wrote:
> >
> > > Jan, Shawn, Susheel
> > >
> > > First steps first. First, let's do a fault-tolerant cluster, then
> maybe a
> > > _geographically_ fault-tolerant cluster.
> > >
> > > Add another server in either DC1 or DC2, in a separate rack, with
> > > independent power etc. As Shawn says below, install the third ZK there.
> > You
> > > would satisfy most of your requirements that way.
> > >
> > > cheers -- Rick
> > >
> > >
> > > On 2017-05-23 12:56 PM, Shawn Heisey wrote:
> > >
> > >> On 5/23/2017 10:12 AM, Susheel Kumar wrote:
> > >>
> > >>> Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster
> > in
> > >>> one of lower env with 6 shards/replica in dc1 & 6 shard/replica in
> dc2
> > >>> (each shard replicated cross data center) with 3 ZK in dc1 and 2 ZK
> in
> > dc2.
> > >>> (I didn't have the availability of 3rd data center for ZK so went
> with
> > only
> > >>> 2 data center with above configuration) and so far no issues. Its
> been
> > >>> running fine, indexing, replicating data, serving queries etc. So in
> my
> > >>> test, setting up single cluster across two zones/data center works
> > without
> > >>> any issue when there is no or very minimal latency (in my case around
> > 30ms
> > >>> one way
> > >>>
> > >>
> > >> With that setup, if dc2 goes down, you're all good, but if dc1 goes
> > down,
> > >> you're not.
> > >>
> > >> There aren't enough ZK servers in dc2 to maintain quorum when dc1 is
> > >> unreachable, and SolrCloud is going to go read-only.  Queries would
> most
> > >> likely work, but you would not be able to change the indexes at all.
> > >>
> > >> ZooKeeper with N total servers requires int((N/2)+1) servers to be
> > >> operational to maintain quorum.  This means that with five total
> > servers,
> > >> three must be operational and able to talk to each other, or ZK cannot
> > >> guarantee that there is no split-brain, so quorum is lost.
> > >>
> > >> ZK in two data centers will never be fully fault-tolerant. There is no
> > >> combination of servers that will work properly.  You must have three
> > data
> > >> centers for a geographically fault-tolerant cluster.  Solr would be
> > >> optional in the third data center.  ZK must be installed in all three.
> > >>
> > >> Thanks,
> > >> Shawn
> > >>
> > >>
> > >
> >
>

Re: Spread SolrCloud across two locations

[Susheel Kumar]

Thanks, Pushkar, Make sense.  Trying to understand the difference between
your setup vs Jan's proposed setup.

- Seems like when DC1 goes down, in your setup we have to bounce *one* from
observer to non-observer while in Jan's setup *two* observers to
non-observers.  Anything else I am missing

- When DC1 comes back -  with your setup we need to bounce the one
non-observer to observer to have 5 nodes quorum otherwise there are 3 + 3
observers while with Jan's setup if we don't take any action when DC1 comes
back, we are still operational with 5 nodes quorum.  Isn't it?  Or I am
missing something.



On Fri, May 26, 2017 at 10:07 AM, Pushkar Raste 
wrote:

> Damn,
> Math is hard
>
> DC1 : 3 non observers
> DC2 : 2 non observers
>
> 3 + 2 = 5 non observers
>
> Observers don't participate in voting = non observers participate in voting
>
> 5 non observers = 5 votes
>
> In addition to the 2 non observer, DC2 also has an observer, which as you
> pointed out does not participate in the voting.
>
> We still have 5 voting nodes.
>
>
> Think of the observer as a standby name node in Hadoop 1.x, where some
> intervention needed if the primary name node went down.
>
>
> I hope my math makes sense
>
> On May 26, 2017 9:04 AM, "Susheel Kumar"  wrote:
>
> From ZK documentation, observers do not participate in vote,  so Pushkar,
> when you said 5 nodes participate in voting, what exactly you mean?
>
> -- Observers are non-voting members of an ensemble which only hear the
> results of votes, not the agreement protocol that leads up to them.
>
> Per ZK documentation, 3.4 includes observers,  does that mean Jan thought
> experiment is practically possible, correct?
>
>
> On Fri, May 26, 2017 at 3:53 AM, Rick Leir  wrote:
>
> > Jan, Shawn, Susheel
> >
> > First steps first. First, let's do a fault-tolerant cluster, then maybe a
> > _geographically_ fault-tolerant cluster.
> >
> > Add another server in either DC1 or DC2, in a separate rack, with
> > independent power etc. As Shawn says below, install the third ZK there.
> You
> > would satisfy most of your requirements that way.
> >
> > cheers -- Rick
> >
> >
> > On 2017-05-23 12:56 PM, Shawn Heisey wrote:
> >
> >> On 5/23/2017 10:12 AM, Susheel Kumar wrote:
> >>
> >>> Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster
> in
> >>> one of lower env with 6 shards/replica in dc1 & 6 shard/replica in dc2
> >>> (each shard replicated cross data center) with 3 ZK in dc1 and 2 ZK in
> dc2.
> >>> (I didn't have the availability of 3rd data center for ZK so went with
> only
> >>> 2 data center with above configuration) and so far no issues. Its been
> >>> running fine, indexing, replicating data, serving queries etc. So in my
> >>> test, setting up single cluster across two zones/data center works
> without
> >>> any issue when there is no or very minimal latency (in my case around
> 30ms
> >>> one way
> >>>
> >>
> >> With that setup, if dc2 goes down, you're all good, but if dc1 goes
> down,
> >> you're not.
> >>
> >> There aren't enough ZK servers in dc2 to maintain quorum when dc1 is
> >> unreachable, and SolrCloud is going to go read-only.  Queries would most
> >> likely work, but you would not be able to change the indexes at all.
> >>
> >> ZooKeeper with N total servers requires int((N/2)+1) servers to be
> >> operational to maintain quorum.  This means that with five total
> servers,
> >> three must be operational and able to talk to each other, or ZK cannot
> >> guarantee that there is no split-brain, so quorum is lost.
> >>
> >> ZK in two data centers will never be fully fault-tolerant. There is no
> >> combination of servers that will work properly.  You must have three
> data
> >> centers for a geographically fault-tolerant cluster.  Solr would be
> >> optional in the third data center.  ZK must be installed in all three.
> >>
> >> Thanks,
> >> Shawn
> >>
> >>
> >
>

Re: Spread SolrCloud across two locations

[Pushkar Raste]

Damn,
Math is hard

DC1 : 3 non observers
DC2 : 2 non observers

3 + 2 = 5 non observers

Observers don't participate in voting = non observers participate in voting

5 non observers = 5 votes

In addition to the 2 non observer, DC2 also has an observer, which as you
pointed out does not participate in the voting.

We still have 5 voting nodes.


Think of the observer as a standby name node in Hadoop 1.x, where some
intervention needed if the primary name node went down.


I hope my math makes sense

On May 26, 2017 9:04 AM, "Susheel Kumar"  wrote:

>From ZK documentation, observers do not participate in vote,  so Pushkar,
when you said 5 nodes participate in voting, what exactly you mean?

-- Observers are non-voting members of an ensemble which only hear the
results of votes, not the agreement protocol that leads up to them.

Per ZK documentation, 3.4 includes observers,  does that mean Jan thought
experiment is practically possible, correct?


On Fri, May 26, 2017 at 3:53 AM, Rick Leir  wrote:

> Jan, Shawn, Susheel
>
> First steps first. First, let's do a fault-tolerant cluster, then maybe a
> _geographically_ fault-tolerant cluster.
>
> Add another server in either DC1 or DC2, in a separate rack, with
> independent power etc. As Shawn says below, install the third ZK there.
You
> would satisfy most of your requirements that way.
>
> cheers -- Rick
>
>
> On 2017-05-23 12:56 PM, Shawn Heisey wrote:
>
>> On 5/23/2017 10:12 AM, Susheel Kumar wrote:
>>
>>> Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster in
>>> one of lower env with 6 shards/replica in dc1 & 6 shard/replica in dc2
>>> (each shard replicated cross data center) with 3 ZK in dc1 and 2 ZK in
dc2.
>>> (I didn't have the availability of 3rd data center for ZK so went with
only
>>> 2 data center with above configuration) and so far no issues. Its been
>>> running fine, indexing, replicating data, serving queries etc. So in my
>>> test, setting up single cluster across two zones/data center works
without
>>> any issue when there is no or very minimal latency (in my case around
30ms
>>> one way
>>>
>>
>> With that setup, if dc2 goes down, you're all good, but if dc1 goes down,
>> you're not.
>>
>> There aren't enough ZK servers in dc2 to maintain quorum when dc1 is
>> unreachable, and SolrCloud is going to go read-only.  Queries would most
>> likely work, but you would not be able to change the indexes at all.
>>
>> ZooKeeper with N total servers requires int((N/2)+1) servers to be
>> operational to maintain quorum.  This means that with five total servers,
>> three must be operational and able to talk to each other, or ZK cannot
>> guarantee that there is no split-brain, so quorum is lost.
>>
>> ZK in two data centers will never be fully fault-tolerant. There is no
>> combination of servers that will work properly.  You must have three data
>> centers for a geographically fault-tolerant cluster.  Solr would be
>> optional in the third data center.  ZK must be installed in all three.
>>
>> Thanks,
>> Shawn
>>
>>
>

Re: Spread SolrCloud across two locations

[Susheel Kumar]

>From ZK documentation, observers do not participate in vote,  so Pushkar,
when you said 5 nodes participate in voting, what exactly you mean?

-- Observers are non-voting members of an ensemble which only hear the
results of votes, not the agreement protocol that leads up to them.

Per ZK documentation, 3.4 includes observers,  does that mean Jan thought
experiment is practically possible, correct?


On Fri, May 26, 2017 at 3:53 AM, Rick Leir  wrote:

> Jan, Shawn, Susheel
>
> First steps first. First, let's do a fault-tolerant cluster, then maybe a
> _geographically_ fault-tolerant cluster.
>
> Add another server in either DC1 or DC2, in a separate rack, with
> independent power etc. As Shawn says below, install the third ZK there. You
> would satisfy most of your requirements that way.
>
> cheers -- Rick
>
>
> On 2017-05-23 12:56 PM, Shawn Heisey wrote:
>
>> On 5/23/2017 10:12 AM, Susheel Kumar wrote:
>>
>>> Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster in
>>> one of lower env with 6 shards/replica in dc1 & 6 shard/replica in dc2
>>> (each shard replicated cross data center) with 3 ZK in dc1 and 2 ZK in dc2.
>>> (I didn't have the availability of 3rd data center for ZK so went with only
>>> 2 data center with above configuration) and so far no issues. Its been
>>> running fine, indexing, replicating data, serving queries etc. So in my
>>> test, setting up single cluster across two zones/data center works without
>>> any issue when there is no or very minimal latency (in my case around 30ms
>>> one way
>>>
>>
>> With that setup, if dc2 goes down, you're all good, but if dc1 goes down,
>> you're not.
>>
>> There aren't enough ZK servers in dc2 to maintain quorum when dc1 is
>> unreachable, and SolrCloud is going to go read-only.  Queries would most
>> likely work, but you would not be able to change the indexes at all.
>>
>> ZooKeeper with N total servers requires int((N/2)+1) servers to be
>> operational to maintain quorum.  This means that with five total servers,
>> three must be operational and able to talk to each other, or ZK cannot
>> guarantee that there is no split-brain, so quorum is lost.
>>
>> ZK in two data centers will never be fully fault-tolerant. There is no
>> combination of servers that will work properly.  You must have three data
>> centers for a geographically fault-tolerant cluster.  Solr would be
>> optional in the third data center.  ZK must be installed in all three.
>>
>> Thanks,
>> Shawn
>>
>>
>

Re: Spread SolrCloud across two locations

[Rick Leir]

Jan, Shawn, Susheel

First steps first. First, let's do a fault-tolerant cluster, then maybe 
a _geographically_ fault-tolerant cluster.


Add another server in either DC1 or DC2, in a separate rack, with 
independent power etc. As Shawn says below, install the third ZK there. 
You would satisfy most of your requirements that way.


cheers -- Rick

On 2017-05-23 12:56 PM, Shawn Heisey wrote:

On 5/23/2017 10:12 AM, Susheel Kumar wrote:
Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster 
in one of lower env with 6 shards/replica in dc1 & 6 shard/replica in 
dc2 (each shard replicated cross data center) with 3 ZK in dc1 and 2 
ZK in dc2. (I didn't have the availability of 3rd data center for ZK 
so went with only 2 data center with above configuration) and so far 
no issues. Its been running fine, indexing, replicating data, serving 
queries etc. So in my test, setting up single cluster across two 
zones/data center works without any issue when there is no or very 
minimal latency (in my case around 30ms one way


With that setup, if dc2 goes down, you're all good, but if dc1 goes 
down, you're not.


There aren't enough ZK servers in dc2 to maintain quorum when dc1 is 
unreachable, and SolrCloud is going to go read-only.  Queries would 
most likely work, but you would not be able to change the indexes at all.


ZooKeeper with N total servers requires int((N/2)+1) servers to be 
operational to maintain quorum.  This means that with five total 
servers, three must be operational and able to talk to each other, or 
ZK cannot guarantee that there is no split-brain, so quorum is lost.


ZK in two data centers will never be fully fault-tolerant. There is no 
combination of servers that will work properly.  You must have three 
data centers for a geographically fault-tolerant cluster.  Solr would 
be optional in the third data center.  ZK must be installed in all three.


Thanks,
Shawn

Re: Spread SolrCloud across two locations

[Pushkar Raste]

ZK 3.5 isn't officially released. It is alpha/beta for years. I wouldn't
use it in production.

The setup I proposed :

DC1 : 3 nodes, all are non observer's.
DC2 : 3 nodes, 2 are non observer's and 1 is observer

This means only 5 nodes participate in voting and 3 nodes make quorum. If
DC1 goes down, you will have to bounce observer and bring it back as non
observer.
This will still cause some down time but it is easier to do than trying to
add another node when you are already in a fire fighting mode.

On May 25, 2017 5:50 PM, "Jan Høydahl"  wrote:

Thanks for the tip Pushkar,

> A setup I have used in the past was to have an observer I  DC2. If DC1 one
I was not aware that ZK 3.4 supports observers, thought it was a 3.5
feature.
So do you setup followers only in DC1 (3x), and observers only in DC2 (3x)
and then point each Solr node to all 6 ZKs?
Then you could get away by flipping the DC2 observers to followers in a
rolling manner and avoid restarting Solr?

> When I tested this solr survived just fine, but it been a while.
Anirudha you say you tested this with 3.5. Does that mean that ZK3.5 works
with Solr?

> Whether ZK 3.5 is there or not, there is potential unknown behavior when
> dc1 comes back online, unless you can have dc1 personnel shut the
> servers down, or block communication between your servers in dc1 and dc2.
Shawn, you are right that in the sketched setup, if DC1 is allowed to come
back up
and join the cluster, then various Solr nodes will use a different ZK
connection string
and the DC1 ZKs won’t talk to all the DC2 ones.

But with the observer model, all nodes know about all ZKs all the time, and
the “odd number of ZKs”
requirement is only applying to the voting followers, see
https://zookeeper.apache.org/doc/trunk/zookeeperObservers.html <
https://zookeeper.apache.org/doc/trunk/zookeeperObservers.html>

Let’s do a thought experiment:
- Have 2 ZKs in DC1, both being followers, and 3 ZKs in DC2, one being
follower and the others observers
- All Solr nodes have a ZK_HOST=zk1dc1,zk2dc1,zk1dc2,zk2dc2,zk3dc2
connection string, i.e. five nodes where initially 3 form the quorum
- At loss of DC2 all is fine
- At loss of DC1, you reconfigure zk2dc2 and zk3dc2 from observers to
followers, meaning they now are in majority (3 of 5) and writes resume
- When DC1 comes back up, zk1dc1 and zk2dc1 are now not a majority and need
to sync up from the others
- After all is stable again, you flip two DC2 ZKs back to observers

I may have misunderstood the observer thing here, but if this is at all
doable, this should be scriptable with ssh or ansible quite easily?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 25. mai 2017 kl. 00.35 skrev Pushkar Raste :
>
> A setup I have used in the past was to have an observer I  DC2. If DC1 one
> goes boom you need manual intervention to change observer's role to make
it
> a follower.
>
> When DC1 comes back up change on instance in DC2 to make it a observer
> again
>
> On May 24, 2017 6:15 PM, "Jan Høydahl"  wrote:
>
>> Sure, ZK does by design not support a two-node/two-location setup. But
>> still, users may want/need to deploy that,
>> and my question was if there are smart ways to make such a setup as
little
>> painful as possible in case of failure.
>>
>> Take the example of DC1: 3xZK and DC2: 2xZK again. And then DC1 goes
BOOM.
>> Without an active action DC2 would be read-only
>> What if then the Ops personnel in DC2 could, with a single
script/command,
>> instruct DC2 to resume “master” role:
>> - Add a 3rd DC2 ZK to the two existing, reconfigure and let them sync up.
>> - Rolling restart of Solr nodes with new ZK_HOST string
>> Of course, they would also then need to make sure that DC1 does not boot
>> up again before compatible change has been done there too.
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>>> 23. mai 2017 kl. 18.56 skrev Shawn Heisey :
>>>
>>> On 5/23/2017 10:12 AM, Susheel Kumar wrote:
 Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster
>> in one of lower env with 6 shards/replica in dc1 & 6 shard/replica in dc2
>> (each shard replicated cross data center) with 3 ZK in dc1 and 2 ZK in
dc2.
>> (I didn't have the availability of 3rd data center for ZK so went with
only
>> 2 data center with above configuration) and so far no issues. Its been
>> running fine, indexing, replicating data, serving queries etc. So in my
>> test, setting up single cluster across two zones/data center works
without
>> any issue when there is no or very minimal latency (in my case around
30ms
>> one way
>>>
>>> With that setup, if dc2 goes down, you're all good, but if dc1 goes
>> down, you're not.
>>>
>>> There aren't enough ZK servers in dc2 to maintain quorum when dc1 is
>> unreachable, and SolrCloud is going to go read-only.  Queries would most
>> likely work, but you would not be able to change the indexes at all.
>>>
>>> ZooKeeper with N total servers requires int((

Re: Spread SolrCloud across two locations

[Jan Høydahl]

Thanks for the tip Pushkar,

> A setup I have used in the past was to have an observer I  DC2. If DC1 one
I was not aware that ZK 3.4 supports observers, thought it was a 3.5 feature.
So do you setup followers only in DC1 (3x), and observers only in DC2 (3x) and 
then point each Solr node to all 6 ZKs?
Then you could get away by flipping the DC2 observers to followers in a rolling 
manner and avoid restarting Solr?

> When I tested this solr survived just fine, but it been a while.
Anirudha you say you tested this with 3.5. Does that mean that ZK3.5 works with 
Solr? 

> Whether ZK 3.5 is there or not, there is potential unknown behavior when
> dc1 comes back online, unless you can have dc1 personnel shut the
> servers down, or block communication between your servers in dc1 and dc2.
Shawn, you are right that in the sketched setup, if DC1 is allowed to come back 
up
and join the cluster, then various Solr nodes will use a different ZK 
connection string
and the DC1 ZKs won’t talk to all the DC2 ones.

But with the observer model, all nodes know about all ZKs all the time, and the 
“odd number of ZKs”
requirement is only applying to the voting followers, see 
https://zookeeper.apache.org/doc/trunk/zookeeperObservers.html 


Let’s do a thought experiment:
- Have 2 ZKs in DC1, both being followers, and 3 ZKs in DC2, one being follower 
and the others observers
- All Solr nodes have a ZK_HOST=zk1dc1,zk2dc1,zk1dc2,zk2dc2,zk3dc2 connection 
string, i.e. five nodes where initially 3 form the quorum
- At loss of DC2 all is fine
- At loss of DC1, you reconfigure zk2dc2 and zk3dc2 from observers to 
followers, meaning they now are in majority (3 of 5) and writes resume
- When DC1 comes back up, zk1dc1 and zk2dc1 are now not a majority and need to 
sync up from the others
- After all is stable again, you flip two DC2 ZKs back to observers

I may have misunderstood the observer thing here, but if this is at all doable, 
this should be scriptable with ssh or ansible quite easily?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 25. mai 2017 kl. 00.35 skrev Pushkar Raste :
> 
> A setup I have used in the past was to have an observer I  DC2. If DC1 one
> goes boom you need manual intervention to change observer's role to make it
> a follower.
> 
> When DC1 comes back up change on instance in DC2 to make it a observer
> again
> 
> On May 24, 2017 6:15 PM, "Jan Høydahl"  wrote:
> 
>> Sure, ZK does by design not support a two-node/two-location setup. But
>> still, users may want/need to deploy that,
>> and my question was if there are smart ways to make such a setup as little
>> painful as possible in case of failure.
>> 
>> Take the example of DC1: 3xZK and DC2: 2xZK again. And then DC1 goes BOOM.
>> Without an active action DC2 would be read-only
>> What if then the Ops personnel in DC2 could, with a single script/command,
>> instruct DC2 to resume “master” role:
>> - Add a 3rd DC2 ZK to the two existing, reconfigure and let them sync up.
>> - Rolling restart of Solr nodes with new ZK_HOST string
>> Of course, they would also then need to make sure that DC1 does not boot
>> up again before compatible change has been done there too.
>> 
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>> 
>>> 23. mai 2017 kl. 18.56 skrev Shawn Heisey :
>>> 
>>> On 5/23/2017 10:12 AM, Susheel Kumar wrote:
 Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster
>> in one of lower env with 6 shards/replica in dc1 & 6 shard/replica in dc2
>> (each shard replicated cross data center) with 3 ZK in dc1 and 2 ZK in dc2.
>> (I didn't have the availability of 3rd data center for ZK so went with only
>> 2 data center with above configuration) and so far no issues. Its been
>> running fine, indexing, replicating data, serving queries etc. So in my
>> test, setting up single cluster across two zones/data center works without
>> any issue when there is no or very minimal latency (in my case around 30ms
>> one way
>>> 
>>> With that setup, if dc2 goes down, you're all good, but if dc1 goes
>> down, you're not.
>>> 
>>> There aren't enough ZK servers in dc2 to maintain quorum when dc1 is
>> unreachable, and SolrCloud is going to go read-only.  Queries would most
>> likely work, but you would not be able to change the indexes at all.
>>> 
>>> ZooKeeper with N total servers requires int((N/2)+1) servers to be
>> operational to maintain quorum.  This means that with five total servers,
>> three must be operational and able to talk to each other, or ZK cannot
>> guarantee that there is no split-brain, so quorum is lost.
>>> 
>>> ZK in two data centers will never be fully fault-tolerant. There is no
>> combination of servers that will work properly.  You must have three data
>> centers for a geographically fault-tolerant cluster.  Solr would be
>> optional in the third data center.  ZK must be installed in all thre

Re: Spread SolrCloud across two locations

[Jan Høydahl]

> Hi - Again, hiring a simple VM at a third location without a Solr cloud 
> sounds like the simplest solution. It keeps the quorum tight and sound. This 
> simple solution is the one i would try first.

My only worry here is latency, and perhaps complexity, security and cost.
- Latency if you have to rent a server in some commercial DC without the same 
fast lines as you have between your enterprise’s own private DCs
- Complexity and cost because you’d need not just some virtual server in some 
public cloud, but most likely a dedicated one with fixed VPN networking etc
- Security could also be a showstopper for certain users who have a requirement 
to self-host things due to data sensitivity, and only have 2 DCs

But if none of these are a concern then go for it!

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

Re: Spread SolrCloud across two locations

[Shawn Heisey]

On 5/24/2017 4:14 PM, Jan Høydahl wrote:
> Sure, ZK does by design not support a two-node/two-location setup. But still, 
> users may want/need to deploy that,
> and my question was if there are smart ways to make such a setup as little 
> painful as possible in case of failure.
>
> Take the example of DC1: 3xZK and DC2: 2xZK again. And then DC1 goes BOOM.
> Without an active action DC2 would be read-only
> What if then the Ops personnel in DC2 could, with a single script/command, 
> instruct DC2 to resume “master” role:
> - Add a 3rd DC2 ZK to the two existing, reconfigure and let them sync up.
> - Rolling restart of Solr nodes with new ZK_HOST string
> Of course, they would also then need to make sure that DC1 does not boot up 
> again before compatible change has been done there too.

When ZK 3.5 comes out and SolrCloud is updated to use it, I think that
it might be possible to remove the dc1 servers from the ensemble and add
another server in dc2 to re-form a new quorum, without restarting
anything.  It could be quite some time before a stable 3.5 is available,
based on past release history.  They don't release anywhere near as
often as Lucene/Solr does.

With the current ZK version, I think your procedure would work, but I
definitely wouldn't call it painless.  Indexing would be unavailable
when dc1 goes down, and everything could be down while the restarts are
happening.

Whether ZK 3.5 is there or not, there is potential unknown behavior when
dc1 comes back online, unless you can have dc1 personnel shut the
servers down, or block communication between your servers in dc1 and dc2.

Overall, having one or two ZK servers in each main DC and a tiebreaker
ZK on a low-cost server in a third DC seems like a better option. 
There's no intervention required when a DC goes down, or when it comes
back up.

Thanks,
Shawn

Re: Spread SolrCloud across two locations

[Anirudha Jadhav]

Latest zk supports auto reconfigure.
Keep one DC as quorum and another as observers.

When a DC goes down initiate a zk reconfigure action. To flip quorum and
observers.

When I tested this solr survived just fine, but it been a while.

Ani


On Wed, May 24, 2017 at 6:35 PM Pushkar Raste 
wrote:

> A setup I have used in the past was to have an observer I  DC2. If DC1 one
> goes boom you need manual intervention to change observer's role to make it
> a follower.
>
> When DC1 comes back up change on instance in DC2 to make it a observer
> again
>
> On May 24, 2017 6:15 PM, "Jan Høydahl"  wrote:
>
> > Sure, ZK does by design not support a two-node/two-location setup. But
> > still, users may want/need to deploy that,
> > and my question was if there are smart ways to make such a setup as
> little
> > painful as possible in case of failure.
> >
> > Take the example of DC1: 3xZK and DC2: 2xZK again. And then DC1 goes
> BOOM.
> > Without an active action DC2 would be read-only
> > What if then the Ops personnel in DC2 could, with a single
> script/command,
> > instruct DC2 to resume “master” role:
> > - Add a 3rd DC2 ZK to the two existing, reconfigure and let them sync up.
> > - Rolling restart of Solr nodes with new ZK_HOST string
> > Of course, they would also then need to make sure that DC1 does not boot
> > up again before compatible change has been done there too.
> >
> > --
> > Jan Høydahl, search solution architect
> > Cominvent AS - www.cominvent.com
> >
> > > 23. mai 2017 kl. 18.56 skrev Shawn Heisey :
> > >
> > > On 5/23/2017 10:12 AM, Susheel Kumar wrote:
> > >> Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster
> > in one of lower env with 6 shards/replica in dc1 & 6 shard/replica in dc2
> > (each shard replicated cross data center) with 3 ZK in dc1 and 2 ZK in
> dc2.
> > (I didn't have the availability of 3rd data center for ZK so went with
> only
> > 2 data center with above configuration) and so far no issues. Its been
> > running fine, indexing, replicating data, serving queries etc. So in my
> > test, setting up single cluster across two zones/data center works
> without
> > any issue when there is no or very minimal latency (in my case around
> 30ms
> > one way
> > >
> > > With that setup, if dc2 goes down, you're all good, but if dc1 goes
> > down, you're not.
> > >
> > > There aren't enough ZK servers in dc2 to maintain quorum when dc1 is
> > unreachable, and SolrCloud is going to go read-only.  Queries would most
> > likely work, but you would not be able to change the indexes at all.
> > >
> > > ZooKeeper with N total servers requires int((N/2)+1) servers to be
> > operational to maintain quorum.  This means that with five total servers,
> > three must be operational and able to talk to each other, or ZK cannot
> > guarantee that there is no split-brain, so quorum is lost.
> > >
> > > ZK in two data centers will never be fully fault-tolerant. There is no
> > combination of servers that will work properly.  You must have three data
> > centers for a geographically fault-tolerant cluster.  Solr would be
> > optional in the third data center.  ZK must be installed in all three.
> > >
> > > Thanks,
> > > Shawn
> > >
> >
> >
>
-- 
Anirudha P. Jadhav

Re: Spread SolrCloud across two locations

[Pushkar Raste]

A setup I have used in the past was to have an observer I  DC2. If DC1 one
goes boom you need manual intervention to change observer's role to make it
a follower.

When DC1 comes back up change on instance in DC2 to make it a observer
again

On May 24, 2017 6:15 PM, "Jan Høydahl"  wrote:

> Sure, ZK does by design not support a two-node/two-location setup. But
> still, users may want/need to deploy that,
> and my question was if there are smart ways to make such a setup as little
> painful as possible in case of failure.
>
> Take the example of DC1: 3xZK and DC2: 2xZK again. And then DC1 goes BOOM.
> Without an active action DC2 would be read-only
> What if then the Ops personnel in DC2 could, with a single script/command,
> instruct DC2 to resume “master” role:
> - Add a 3rd DC2 ZK to the two existing, reconfigure and let them sync up.
> - Rolling restart of Solr nodes with new ZK_HOST string
> Of course, they would also then need to make sure that DC1 does not boot
> up again before compatible change has been done there too.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 23. mai 2017 kl. 18.56 skrev Shawn Heisey :
> >
> > On 5/23/2017 10:12 AM, Susheel Kumar wrote:
> >> Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster
> in one of lower env with 6 shards/replica in dc1 & 6 shard/replica in dc2
> (each shard replicated cross data center) with 3 ZK in dc1 and 2 ZK in dc2.
> (I didn't have the availability of 3rd data center for ZK so went with only
> 2 data center with above configuration) and so far no issues. Its been
> running fine, indexing, replicating data, serving queries etc. So in my
> test, setting up single cluster across two zones/data center works without
> any issue when there is no or very minimal latency (in my case around 30ms
> one way
> >
> > With that setup, if dc2 goes down, you're all good, but if dc1 goes
> down, you're not.
> >
> > There aren't enough ZK servers in dc2 to maintain quorum when dc1 is
> unreachable, and SolrCloud is going to go read-only.  Queries would most
> likely work, but you would not be able to change the indexes at all.
> >
> > ZooKeeper with N total servers requires int((N/2)+1) servers to be
> operational to maintain quorum.  This means that with five total servers,
> three must be operational and able to talk to each other, or ZK cannot
> guarantee that there is no split-brain, so quorum is lost.
> >
> > ZK in two data centers will never be fully fault-tolerant. There is no
> combination of servers that will work properly.  You must have three data
> centers for a geographically fault-tolerant cluster.  Solr would be
> optional in the third data center.  ZK must be installed in all three.
> >
> > Thanks,
> > Shawn
> >
>
>

RE: Spread SolrCloud across two locations

[Markus Jelsma]

Hi - Again, hiring a simple VM at a third location without a Solr cloud sounds 
like the simplest solution. It keeps the quorum tight and sound. This simple 
solution is the one i would try first.

Or am i completely missing something and sound like an idiot? Could be, of 
course.

Regards,
Markus

 
 
-Original message-
> From:Jan Høydahl 
> Sent: Thursday 25th May 2017 0:15
> To: solr-user@lucene.apache.org
> Subject: Re: Spread SolrCloud across two locations
> 
> Sure, ZK does by design not support a two-node/two-location setup. But still, 
> users may want/need to deploy that,
> and my question was if there are smart ways to make such a setup as little 
> painful as possible in case of failure.
> 
> Take the example of DC1: 3xZK and DC2: 2xZK again. And then DC1 goes BOOM.
> Without an active action DC2 would be read-only
> What if then the Ops personnel in DC2 could, with a single script/command, 
> instruct DC2 to resume “master” role:
> - Add a 3rd DC2 ZK to the two existing, reconfigure and let them sync up.
> - Rolling restart of Solr nodes with new ZK_HOST string
> Of course, they would also then need to make sure that DC1 does not boot up 
> again before compatible change has been done there too.
> 
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
> 
> > 23. mai 2017 kl. 18.56 skrev Shawn Heisey :
> > 
> > On 5/23/2017 10:12 AM, Susheel Kumar wrote:
> >> Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster in 
> >> one of lower env with 6 shards/replica in dc1 & 6 shard/replica in dc2 
> >> (each shard replicated cross data center) with 3 ZK in dc1 and 2 ZK in 
> >> dc2. (I didn't have the availability of 3rd data center for ZK so went 
> >> with only 2 data center with above configuration) and so far no issues. 
> >> Its been running fine, indexing, replicating data, serving queries etc. So 
> >> in my test, setting up single cluster across two zones/data center works 
> >> without any issue when there is no or very minimal latency (in my case 
> >> around 30ms one way
> > 
> > With that setup, if dc2 goes down, you're all good, but if dc1 goes down, 
> > you're not.
> > 
> > There aren't enough ZK servers in dc2 to maintain quorum when dc1 is 
> > unreachable, and SolrCloud is going to go read-only.  Queries would most 
> > likely work, but you would not be able to change the indexes at all.
> > 
> > ZooKeeper with N total servers requires int((N/2)+1) servers to be 
> > operational to maintain quorum.  This means that with five total servers, 
> > three must be operational and able to talk to each other, or ZK cannot 
> > guarantee that there is no split-brain, so quorum is lost.
> > 
> > ZK in two data centers will never be fully fault-tolerant. There is no 
> > combination of servers that will work properly.  You must have three data 
> > centers for a geographically fault-tolerant cluster.  Solr would be 
> > optional in the third data center.  ZK must be installed in all three.
> > 
> > Thanks,
> > Shawn
> > 
> 
> 

Re: Spread SolrCloud across two locations

[Jan Høydahl]

Sure, ZK does by design not support a two-node/two-location setup. But still, 
users may want/need to deploy that,
and my question was if there are smart ways to make such a setup as little 
painful as possible in case of failure.

Take the example of DC1: 3xZK and DC2: 2xZK again. And then DC1 goes BOOM.
Without an active action DC2 would be read-only
What if then the Ops personnel in DC2 could, with a single script/command, 
instruct DC2 to resume “master” role:
- Add a 3rd DC2 ZK to the two existing, reconfigure and let them sync up.
- Rolling restart of Solr nodes with new ZK_HOST string
Of course, they would also then need to make sure that DC1 does not boot up 
again before compatible change has been done there too.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 23. mai 2017 kl. 18.56 skrev Shawn Heisey :
> 
> On 5/23/2017 10:12 AM, Susheel Kumar wrote:
>> Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster in one 
>> of lower env with 6 shards/replica in dc1 & 6 shard/replica in dc2 (each 
>> shard replicated cross data center) with 3 ZK in dc1 and 2 ZK in dc2. (I 
>> didn't have the availability of 3rd data center for ZK so went with only 2 
>> data center with above configuration) and so far no issues. Its been running 
>> fine, indexing, replicating data, serving queries etc. So in my test, 
>> setting up single cluster across two zones/data center works without any 
>> issue when there is no or very minimal latency (in my case around 30ms one 
>> way
> 
> With that setup, if dc2 goes down, you're all good, but if dc1 goes down, 
> you're not.
> 
> There aren't enough ZK servers in dc2 to maintain quorum when dc1 is 
> unreachable, and SolrCloud is going to go read-only.  Queries would most 
> likely work, but you would not be able to change the indexes at all.
> 
> ZooKeeper with N total servers requires int((N/2)+1) servers to be 
> operational to maintain quorum.  This means that with five total servers, 
> three must be operational and able to talk to each other, or ZK cannot 
> guarantee that there is no split-brain, so quorum is lost.
> 
> ZK in two data centers will never be fully fault-tolerant. There is no 
> combination of servers that will work properly.  You must have three data 
> centers for a geographically fault-tolerant cluster.  Solr would be optional 
> in the third data center.  ZK must be installed in all three.
> 
> Thanks,
> Shawn
> 

Re: Spread SolrCloud across two locations

[Shawn Heisey]

On 5/23/2017 10:12 AM, Susheel Kumar wrote:
Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster 
in one of lower env with 6 shards/replica in dc1 & 6 shard/replica in 
dc2 (each shard replicated cross data center) with 3 ZK in dc1 and 2 
ZK in dc2. (I didn't have the availability of 3rd data center for ZK 
so went with only 2 data center with above configuration) and so far 
no issues. Its been running fine, indexing, replicating data, serving 
queries etc. So in my test, setting up single cluster across two 
zones/data center works without any issue when there is no or very 
minimal latency (in my case around 30ms one way


With that setup, if dc2 goes down, you're all good, but if dc1 goes 
down, you're not.


There aren't enough ZK servers in dc2 to maintain quorum when dc1 is 
unreachable, and SolrCloud is going to go read-only.  Queries would most 
likely work, but you would not be able to change the indexes at all.


ZooKeeper with N total servers requires int((N/2)+1) servers to be 
operational to maintain quorum.  This means that with five total 
servers, three must be operational and able to talk to each other, or ZK 
cannot guarantee that there is no split-brain, so quorum is lost.


ZK in two data centers will never be fully fault-tolerant. There is no 
combination of servers that will work properly.  You must have three 
data centers for a geographically fault-tolerant cluster.  Solr would be 
optional in the third data center.  ZK must be installed in all three.


Thanks,
Shawn

Re: Spread SolrCloud across two locations

[Susheel Kumar]

Agree, Erick.  Since this setup is in our test env, haven't really invested
to add another DC but for Prod sure, will go by DC3 if we do go with this
setup.

On Tue, May 23, 2017 at 12:38 PM, Erick Erickson 
wrote:

> Susheel:
>
> The issue is that if, for any reason at all, the connection between
> dc1 and dc2 is broken, there will be no indexing on dc2 since the Solr
> servers there will not sense ZK quorum. You'll have to do something
> manual to reconfigure.
>
> That's not a flaw in your setup, just the way things work ;).
>
> Putting one of the ZKs on a third DC would change that...
>
> Best,
> Erick
>
> On Tue, May 23, 2017 at 9:12 AM, Susheel Kumar 
> wrote:
> > Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster in
> > one of lower env with 6 shards/replica in dc1 & 6 shard/replica in dc2
> (each
> > shard replicated cross data center) with 3 ZK in dc1 and 2 ZK in dc2.  (I
> > didn't have the availability of 3rd data center for ZK so went with only
> 2
> > data center with above configuration)  and so far no issues.  Its been
> > running fine, indexing, replicating data, serving queries etc. So in my
> > test, setting up single cluster across two zones/data center works
> without
> > any issue when there is no or very minimal latency (in my case around
> 30ms
> > one way)
> >
> > Thanks,
> > Susheel
> >
> > On Tue, May 23, 2017 at 9:20 AM, Jan Høydahl 
> wrote:
> >
> >> I.e. tell the customer that in order to have automatic failover and
> >> recovery in a 2-location setup we require at least one ZK instance in a
> >> separate third location. Kind of a tough requirement but necessary to
> >> safe-guard against split brain during network partition.
> >>
> >> If a third location is not an option, how would you setup ZK for manual
> >> reconfiguration?
> >> Two ZK in DC1 and one in DC2 would give you automatic recovery in case
> DC2
> >> falls out, but if DC1 falls out, WRITE would be disabled and to resume
> >> write in DC2 only, one would need to stop Solr + ZK, reconfigure ZK in
> DC2
> >> as standalone (or setup two more) and then start Solr again with only
> one
> >> ZK.
> >>
> >> --
> >> Jan Høydahl, search solution architect
> >> Cominvent AS - www.cominvent.com
> >>
> >> > 23. mai 2017 kl. 11.14 skrev Markus Jelsma <
> markus.jel...@openindex.io>:
> >> >
> >> > I would probably start by renting a VM at a third location to run
> >> Zookeeper.
> >> >
> >> > Markus
> >> >
> >> > -Original message-
> >> >> From:Jan Høydahl 
> >> >> Sent: Tuesday 23rd May 2017 11:09
> >> >> To: solr-user 
> >> >> Subject: Spread SolrCloud across two locations
> >> >>
> >> >> Hi,
> >> >>
> >> >> A customer has two locations (a few km apart) with super-fast
> >> networking in-between, so for day-to-day operation they view all VMs in
> >> both locations as a pool of servers. They typically spin up redundant
> >> servers for various services in each zone and if a zone should fail
> (they
> >> are a few km apart), the other will just continue working.
> >> >>
> >> >> How can we best support such a setup with Cloud and Zookeeper?
> >> >> They do not need (or want) CDCR since latency and bandwidth is no
> >> problem, and CDCR is active-passive only so it anyway requires manual
> >> intervention to catch up if indexing is switched to the passive DC
> >> temporarily.
> >> >> If it was not for ZK I would setup one Cloud cluster and make sure
> each
> >> shard was replicated cross zones and all would be fine.
> >> >> But ZK really requires a third location in order to tolerate loss of
> an
> >> entire location/zone.
> >> >> All solutions I can think of involves manual intervention,
> >> re-configuring of ZK followed by a restart of the surviving Solr nodes
> in
> >> order to point to the “new” ZK.
> >> >>
> >> >> How have you guys solved such setups?
> >> >>
> >> >> --
> >> >> Jan Høydahl, search solution architect
> >> >> Cominvent AS - www.cominvent.com
> >> >>
> >> >>
> >>
> >>
>

Re: Spread SolrCloud across two locations

[Erick Erickson]

Susheel:

The issue is that if, for any reason at all, the connection between
dc1 and dc2 is broken, there will be no indexing on dc2 since the Solr
servers there will not sense ZK quorum. You'll have to do something
manual to reconfigure.

That's not a flaw in your setup, just the way things work ;).

Putting one of the ZKs on a third DC would change that...

Best,
Erick

On Tue, May 23, 2017 at 9:12 AM, Susheel Kumar  wrote:
> Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster in
> one of lower env with 6 shards/replica in dc1 & 6 shard/replica in dc2 (each
> shard replicated cross data center) with 3 ZK in dc1 and 2 ZK in dc2.  (I
> didn't have the availability of 3rd data center for ZK so went with only 2
> data center with above configuration)  and so far no issues.  Its been
> running fine, indexing, replicating data, serving queries etc. So in my
> test, setting up single cluster across two zones/data center works without
> any issue when there is no or very minimal latency (in my case around 30ms
> one way)
>
> Thanks,
> Susheel
>
> On Tue, May 23, 2017 at 9:20 AM, Jan Høydahl  wrote:
>
>> I.e. tell the customer that in order to have automatic failover and
>> recovery in a 2-location setup we require at least one ZK instance in a
>> separate third location. Kind of a tough requirement but necessary to
>> safe-guard against split brain during network partition.
>>
>> If a third location is not an option, how would you setup ZK for manual
>> reconfiguration?
>> Two ZK in DC1 and one in DC2 would give you automatic recovery in case DC2
>> falls out, but if DC1 falls out, WRITE would be disabled and to resume
>> write in DC2 only, one would need to stop Solr + ZK, reconfigure ZK in DC2
>> as standalone (or setup two more) and then start Solr again with only one
>> ZK.
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>> > 23. mai 2017 kl. 11.14 skrev Markus Jelsma :
>> >
>> > I would probably start by renting a VM at a third location to run
>> Zookeeper.
>> >
>> > Markus
>> >
>> > -Original message-
>> >> From:Jan Høydahl 
>> >> Sent: Tuesday 23rd May 2017 11:09
>> >> To: solr-user 
>> >> Subject: Spread SolrCloud across two locations
>> >>
>> >> Hi,
>> >>
>> >> A customer has two locations (a few km apart) with super-fast
>> networking in-between, so for day-to-day operation they view all VMs in
>> both locations as a pool of servers. They typically spin up redundant
>> servers for various services in each zone and if a zone should fail (they
>> are a few km apart), the other will just continue working.
>> >>
>> >> How can we best support such a setup with Cloud and Zookeeper?
>> >> They do not need (or want) CDCR since latency and bandwidth is no
>> problem, and CDCR is active-passive only so it anyway requires manual
>> intervention to catch up if indexing is switched to the passive DC
>> temporarily.
>> >> If it was not for ZK I would setup one Cloud cluster and make sure each
>> shard was replicated cross zones and all would be fine.
>> >> But ZK really requires a third location in order to tolerate loss of an
>> entire location/zone.
>> >> All solutions I can think of involves manual intervention,
>> re-configuring of ZK followed by a restart of the surviving Solr nodes in
>> order to point to the “new” ZK.
>> >>
>> >> How have you guys solved such setups?
>> >>
>> >> --
>> >> Jan Høydahl, search solution architect
>> >> Cominvent AS - www.cominvent.com
>> >>
>> >>
>>
>>

Re: Spread SolrCloud across two locations

[Susheel Kumar]

Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster in
one of lower env with 6 shards/replica in dc1 & 6 shard/replica in dc2 (each
shard replicated cross data center) with 3 ZK in dc1 and 2 ZK in dc2.  (I
didn't have the availability of 3rd data center for ZK so went with only 2
data center with above configuration)  and so far no issues.  Its been
running fine, indexing, replicating data, serving queries etc. So in my
test, setting up single cluster across two zones/data center works without
any issue when there is no or very minimal latency (in my case around 30ms
one way)

Thanks,
Susheel

On Tue, May 23, 2017 at 9:20 AM, Jan Høydahl  wrote:

> I.e. tell the customer that in order to have automatic failover and
> recovery in a 2-location setup we require at least one ZK instance in a
> separate third location. Kind of a tough requirement but necessary to
> safe-guard against split brain during network partition.
>
> If a third location is not an option, how would you setup ZK for manual
> reconfiguration?
> Two ZK in DC1 and one in DC2 would give you automatic recovery in case DC2
> falls out, but if DC1 falls out, WRITE would be disabled and to resume
> write in DC2 only, one would need to stop Solr + ZK, reconfigure ZK in DC2
> as standalone (or setup two more) and then start Solr again with only one
> ZK.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 23. mai 2017 kl. 11.14 skrev Markus Jelsma :
> >
> > I would probably start by renting a VM at a third location to run
> Zookeeper.
> >
> > Markus
> >
> > -Original message-
> >> From:Jan Høydahl 
> >> Sent: Tuesday 23rd May 2017 11:09
> >> To: solr-user 
> >> Subject: Spread SolrCloud across two locations
> >>
> >> Hi,
> >>
> >> A customer has two locations (a few km apart) with super-fast
> networking in-between, so for day-to-day operation they view all VMs in
> both locations as a pool of servers. They typically spin up redundant
> servers for various services in each zone and if a zone should fail (they
> are a few km apart), the other will just continue working.
> >>
> >> How can we best support such a setup with Cloud and Zookeeper?
> >> They do not need (or want) CDCR since latency and bandwidth is no
> problem, and CDCR is active-passive only so it anyway requires manual
> intervention to catch up if indexing is switched to the passive DC
> temporarily.
> >> If it was not for ZK I would setup one Cloud cluster and make sure each
> shard was replicated cross zones and all would be fine.
> >> But ZK really requires a third location in order to tolerate loss of an
> entire location/zone.
> >> All solutions I can think of involves manual intervention,
> re-configuring of ZK followed by a restart of the surviving Solr nodes in
> order to point to the “new” ZK.
> >>
> >> How have you guys solved such setups?
> >>
> >> --
> >> Jan Høydahl, search solution architect
> >> Cominvent AS - www.cominvent.com
> >>
> >>
>
>

Re: Spread SolrCloud across two locations

[Jan Høydahl]

I.e. tell the customer that in order to have automatic failover and recovery in 
a 2-location setup we require at least one ZK instance in a separate third 
location. Kind of a tough requirement but necessary to safe-guard against split 
brain during network partition. 

If a third location is not an option, how would you setup ZK for manual 
reconfiguration? 
Two ZK in DC1 and one in DC2 would give you automatic recovery in case DC2 
falls out, but if DC1 falls out, WRITE would be disabled and to resume write in 
DC2 only, one would need to stop Solr + ZK, reconfigure ZK in DC2 as standalone 
(or setup two more) and then start Solr again with only one ZK.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 23. mai 2017 kl. 11.14 skrev Markus Jelsma :
> 
> I would probably start by renting a VM at a third location to run Zookeeper.
> 
> Markus
> 
> -Original message-
>> From:Jan Høydahl 
>> Sent: Tuesday 23rd May 2017 11:09
>> To: solr-user 
>> Subject: Spread SolrCloud across two locations
>> 
>> Hi,
>> 
>> A customer has two locations (a few km apart) with super-fast networking 
>> in-between, so for day-to-day operation they view all VMs in both locations 
>> as a pool of servers. They typically spin up redundant servers for various 
>> services in each zone and if a zone should fail (they are a few km apart), 
>> the other will just continue working.
>> 
>> How can we best support such a setup with Cloud and Zookeeper? 
>> They do not need (or want) CDCR since latency and bandwidth is no problem, 
>> and CDCR is active-passive only so it anyway requires manual intervention to 
>> catch up if indexing is switched to the passive DC temporarily.
>> If it was not for ZK I would setup one Cloud cluster and make sure each 
>> shard was replicated cross zones and all would be fine.
>> But ZK really requires a third location in order to tolerate loss of an 
>> entire location/zone.
>> All solutions I can think of involves manual intervention, re-configuring of 
>> ZK followed by a restart of the surviving Solr nodes in order to point to 
>> the “new” ZK.
>> 
>> How have you guys solved such setups?
>> 
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>> 
>> 

RE: Spread SolrCloud across two locations

[Markus Jelsma]

I would probably start by renting a VM at a third location to run Zookeeper.

Markus
 
-Original message-
> From:Jan Høydahl 
> Sent: Tuesday 23rd May 2017 11:09
> To: solr-user 
> Subject: Spread SolrCloud across two locations
> 
> Hi,
> 
> A customer has two locations (a few km apart) with super-fast networking 
> in-between, so for day-to-day operation they view all VMs in both locations 
> as a pool of servers. They typically spin up redundant servers for various 
> services in each zone and if a zone should fail (they are a few km apart), 
> the other will just continue working.
> 
> How can we best support such a setup with Cloud and Zookeeper? 
> They do not need (or want) CDCR since latency and bandwidth is no problem, 
> and CDCR is active-passive only so it anyway requires manual intervention to 
> catch up if indexing is switched to the passive DC temporarily.
> If it was not for ZK I would setup one Cloud cluster and make sure each shard 
> was replicated cross zones and all would be fine.
> But ZK really requires a third location in order to tolerate loss of an 
> entire location/zone.
> All solutions I can think of involves manual intervention, re-configuring of 
> ZK followed by a restart of the surviving Solr nodes in order to point to the 
> “new” ZK.
> 
> How have you guys solved such setups?
> 
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
> 
>