Re: Copying a SolrCloud collection to other hosts

[Jeff Wartes]

I really like the fetchindex approach. Once I figured out the undocumented API, 
it worked really well, and I haven't had to change my usage for any Solr I've 
tried between 4.7-7.2. I recall having some issues if I tried to apply a 
fetchindex to a shard that already had data, where it'd get confused about 
whether it was already newer. But if you're using it against a clean index, 
it's pretty slick, and about the only way to cleanly copy a collection to a 
different cluster (using a different ZK) without a shared filesystem. 

In my tool, I rigged it so that if you were copying into a new collection with 
RF > 1, it'd copy to the replica1's in the new collection from the old 
collection, but then copy the replica2 from the replica1 in the *new* 
collection, saving on theoretical cross-cluster bandwidth, which is the CDCR 
concern too. (fetchindex doesn't apparently trigger the standard replication, 
since it doesn't go through the tlog)

In short, if the target collection is getting writes during this process, 
you're probably going to have issues. If not though, I agree it'd be pretty 
simple to set up. You just need to make sure the destination cluster can talk 
to the source cluster, and that the target collection has the same shard count 
and routing strategy.


On 3/28/18, 10:06 AM, "Erick Erickson"  wrote:

Hmmm, wouldn't even be all that hard would it? A collections API call.

Assuming both collection's state.json nodes were available from ZooKeeper
a command would have all the necessary information, only an HTTP connection
required. I don't think it would be too much of a stretch to be able to 
provide
the other collection's ZooKeeper ensemble if they were different.

I used "other collection" purposely here, among the many details to be
worked out would be whether this was run against the source or target
collection, how all the replicas on the target collection would be
updated..

There may even be infrastructure already in place for CDCR that could
be leveraged since the bootstrapping already does this, even across
data centers.

Not that I'm going to have time to work on it in any near-term though.

On Wed, Mar 28, 2018 at 9:53 AM, David Smiley  
wrote:
> Right, there is a shared filesystem requirement.  It would be nice if this
> Solr feature could be enhanced to have more options like backing up
> directly to another SolrCloud using replication/fetchIndex like your cool
> solrcloud_manager thing.
>
> On Wed, Mar 28, 2018 at 12:34 PM Jeff Wartes  
wrote:
>
>> The backup/restore still requires setting up a shared filesystem on all
>> your nodes though right?
>>
>> I've been using the fetchindex trick in my solrcloud_manager tool for 
ages
>> now: https://github.com/whitepages/solrcloud_manager#cluster-commands
>> Some of the original features in that tool have been incorporated into
>> Solr itself these days, but I still use clonecollection/copycollection
>> regularly. (most recently with Solr 7.2)
>>
>>
>> On 3/27/18, 9:55 PM, "David Smiley"  wrote:
>>
>> The backup/restore API is intended to address this.
>>
>> 
https://builds.apache.org/job/Solr-reference-guide-master/javadoc/making-and-restoring-backups.html
>>
>> Erick's advice is good (and I once drafted docs for the same scheme
>> years
>> ago as well), but I consider it dated -- it's what people had to do
>> before
>> the backup/restore API existed.  Internally, backup/restore is doing
>> similar stuff.  It's easy to give backup/restore a try; surely you
>> have by
>> now?
>>
>> ~ David
>>
>> On Tue, Mar 6, 2018 at 9:47 AM Patrick Schemitz  
wrote:
>>
>> > Hi List,
>> >
>> > so I'm running a bunch of SolrCloud clusters (each cluster is: 8
>> shards
>> > on 2 servers, with 4 instances per server, no replicas, i.e. 1 
shard
>> per
>> > instance).
>> >
>> > Building the index afresh takes 15+ hours, so when I have to deploy
>> a new
>> > index, I build it once, on one cluster, and then copy (scp) over 
the
>> > data//index directories (shutting down the Solr 
instances
>> > first).
>> >
>> > I could get Solr 6.5.1 to number the shard/replica directories
>> nicely via
>> > the createNodeSet and createNodeSet.shuffle options:
>> >
>> > Solr 6.5.1 /var/lib/solr:
>> >
>> > Server node 1:
>> > instance00/data/main_index_shard1_replica1
>> > instance01/data/main_index_shard2_replica1
>> > instance02/data/main_index_shard3_replica1
>> > instance03/data/main_index_shard4_replica1
>> >
   

Re: Copying a SolrCloud collection to other hosts

[Jeff Wartes]

It'd work fine, I played around with this a bit once upon a time. 

The trick is that you need to either:
1. Make sure all shards for the index are synchronized onto every node for the 
duration of the restore (as you mention)
2. Know exactly which nodes will ask to restore which shards for the duration 
of the restore

But the former isn't tenable if you're sharding due to space constraints, and 
the latter can't be easily predicted.


On 3/28/18, 11:30 AM, "Shawn Heisey"  wrote:

On 3/28/2018 10:34 AM, Jeff Wartes wrote:
> The backup/restore still requires setting up a shared filesystem on all 
your nodes though right? 

Technically speaking, I don't think a shared filesystem is actually
REQUIRED to make a backup.

But in order to do a restore, all machines involved with the restore
must have access to all of the data written by all of the machines that
made the backup.

The easiest way to accommodate the restore requirements is to have all
the servers use the same network-mounted filesystem, mounted in the same
location.

One thing that I think you COULD do is have each server make its backup
into a local filesystem, and then use some kind of synchronization tool
(rsync, robocopy, and others) to ensure that every system has an
identical copy of ALL the data, and allow that synchronization to finish
before attempting a restore.  It could be challenging to do the
synchronization successfully without accidentally deleting part of the
data.  The shared filesystem is likely to be easier.

Someone who's more familiar with the feature should comment on whether
that synchronization idea will actually work.  If the backup/restore
functionality is written the way I would *expect* it to be written, it
should work.

Thanks,
Shawn

Re: Copying a SolrCloud collection to other hosts

[Shawn Heisey]

On 3/28/2018 10:34 AM, Jeff Wartes wrote:
> The backup/restore still requires setting up a shared filesystem on all your 
> nodes though right? 

Technically speaking, I don't think a shared filesystem is actually
REQUIRED to make a backup.

But in order to do a restore, all machines involved with the restore
must have access to all of the data written by all of the machines that
made the backup.

The easiest way to accommodate the restore requirements is to have all
the servers use the same network-mounted filesystem, mounted in the same
location.

One thing that I think you COULD do is have each server make its backup
into a local filesystem, and then use some kind of synchronization tool
(rsync, robocopy, and others) to ensure that every system has an
identical copy of ALL the data, and allow that synchronization to finish
before attempting a restore.  It could be challenging to do the
synchronization successfully without accidentally deleting part of the
data.  The shared filesystem is likely to be easier.

Someone who's more familiar with the feature should comment on whether
that synchronization idea will actually work.  If the backup/restore
functionality is written the way I would *expect* it to be written, it
should work.

Thanks,
Shawn

Re: Copying a SolrCloud collection to other hosts

[Erick Erickson]

Hmmm, wouldn't even be all that hard would it? A collections API call.

Assuming both collection's state.json nodes were available from ZooKeeper
a command would have all the necessary information, only an HTTP connection
required. I don't think it would be too much of a stretch to be able to provide
the other collection's ZooKeeper ensemble if they were different.

I used "other collection" purposely here, among the many details to be
worked out would be whether this was run against the source or target
collection, how all the replicas on the target collection would be
updated..

There may even be infrastructure already in place for CDCR that could
be leveraged since the bootstrapping already does this, even across
data centers.

Not that I'm going to have time to work on it in any near-term though.

On Wed, Mar 28, 2018 at 9:53 AM, David Smiley  wrote:
> Right, there is a shared filesystem requirement.  It would be nice if this
> Solr feature could be enhanced to have more options like backing up
> directly to another SolrCloud using replication/fetchIndex like your cool
> solrcloud_manager thing.
>
> On Wed, Mar 28, 2018 at 12:34 PM Jeff Wartes  wrote:
>
>> The backup/restore still requires setting up a shared filesystem on all
>> your nodes though right?
>>
>> I've been using the fetchindex trick in my solrcloud_manager tool for ages
>> now: https://github.com/whitepages/solrcloud_manager#cluster-commands
>> Some of the original features in that tool have been incorporated into
>> Solr itself these days, but I still use clonecollection/copycollection
>> regularly. (most recently with Solr 7.2)
>>
>>
>> On 3/27/18, 9:55 PM, "David Smiley"  wrote:
>>
>> The backup/restore API is intended to address this.
>>
>> https://builds.apache.org/job/Solr-reference-guide-master/javadoc/making-and-restoring-backups.html
>>
>> Erick's advice is good (and I once drafted docs for the same scheme
>> years
>> ago as well), but I consider it dated -- it's what people had to do
>> before
>> the backup/restore API existed.  Internally, backup/restore is doing
>> similar stuff.  It's easy to give backup/restore a try; surely you
>> have by
>> now?
>>
>> ~ David
>>
>> On Tue, Mar 6, 2018 at 9:47 AM Patrick Schemitz  wrote:
>>
>> > Hi List,
>> >
>> > so I'm running a bunch of SolrCloud clusters (each cluster is: 8
>> shards
>> > on 2 servers, with 4 instances per server, no replicas, i.e. 1 shard
>> per
>> > instance).
>> >
>> > Building the index afresh takes 15+ hours, so when I have to deploy
>> a new
>> > index, I build it once, on one cluster, and then copy (scp) over the
>> > data//index directories (shutting down the Solr instances
>> > first).
>> >
>> > I could get Solr 6.5.1 to number the shard/replica directories
>> nicely via
>> > the createNodeSet and createNodeSet.shuffle options:
>> >
>> > Solr 6.5.1 /var/lib/solr:
>> >
>> > Server node 1:
>> > instance00/data/main_index_shard1_replica1
>> > instance01/data/main_index_shard2_replica1
>> > instance02/data/main_index_shard3_replica1
>> > instance03/data/main_index_shard4_replica1
>> >
>> > Server node 2:
>> > instance00/data/main_index_shard5_replica1
>> > instance01/data/main_index_shard6_replica1
>> > instance02/data/main_index_shard7_replica1
>> > instance03/data/main_index_shard8_replica1
>> >
>> > However, while attempting to upgrade to 7.2.1, this numbering has
>> changed:
>> >
>> > Solr 7.2.1 /var/lib/solr:
>> >
>> > Server node 1:
>> > instance00/data/main_index_shard1_replica_n1
>> > instance01/data/main_index_shard2_replica_n2
>> > instance02/data/main_index_shard3_replica_n4
>> > instance03/data/main_index_shard4_replica_n6
>> >
>> > Server node 2:
>> > instance00/data/main_index_shard5_replica_n8
>> > instance01/data/main_index_shard6_replica_n10
>> > instance02/data/main_index_shard7_replica_n12
>> > instance03/data/main_index_shard8_replica_n14
>> >
>> > This new numbering breaks my copy script, and furthermode, I'm
>> worried
>> > as to what happens when the numbering is different among target
>> clusters.
>> >
>> > How can I switch this back to the old numbering scheme?
>> >
>> > Side note: is there a recommended way of doing this? Is the
>> > backup/restore mechanism suitable for this? The ref guide is kind of
>> terse
>> > here.
>> >
>> > Thanks in advance,
>> >
>> > Ciao, Patrick
>> >
>> --
>> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
>> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
>> http://www.solrenterprisesearchserver.com
>>
>>
>> --
> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
> LinkedIn: 

Re: Copying a SolrCloud collection to other hosts

[David Smiley]

Right, there is a shared filesystem requirement.  It would be nice if this
Solr feature could be enhanced to have more options like backing up
directly to another SolrCloud using replication/fetchIndex like your cool
solrcloud_manager thing.

On Wed, Mar 28, 2018 at 12:34 PM Jeff Wartes  wrote:

> The backup/restore still requires setting up a shared filesystem on all
> your nodes though right?
>
> I've been using the fetchindex trick in my solrcloud_manager tool for ages
> now: https://github.com/whitepages/solrcloud_manager#cluster-commands
> Some of the original features in that tool have been incorporated into
> Solr itself these days, but I still use clonecollection/copycollection
> regularly. (most recently with Solr 7.2)
>
>
> On 3/27/18, 9:55 PM, "David Smiley"  wrote:
>
> The backup/restore API is intended to address this.
>
> https://builds.apache.org/job/Solr-reference-guide-master/javadoc/making-and-restoring-backups.html
>
> Erick's advice is good (and I once drafted docs for the same scheme
> years
> ago as well), but I consider it dated -- it's what people had to do
> before
> the backup/restore API existed.  Internally, backup/restore is doing
> similar stuff.  It's easy to give backup/restore a try; surely you
> have by
> now?
>
> ~ David
>
> On Tue, Mar 6, 2018 at 9:47 AM Patrick Schemitz  wrote:
>
> > Hi List,
> >
> > so I'm running a bunch of SolrCloud clusters (each cluster is: 8
> shards
> > on 2 servers, with 4 instances per server, no replicas, i.e. 1 shard
> per
> > instance).
> >
> > Building the index afresh takes 15+ hours, so when I have to deploy
> a new
> > index, I build it once, on one cluster, and then copy (scp) over the
> > data//index directories (shutting down the Solr instances
> > first).
> >
> > I could get Solr 6.5.1 to number the shard/replica directories
> nicely via
> > the createNodeSet and createNodeSet.shuffle options:
> >
> > Solr 6.5.1 /var/lib/solr:
> >
> > Server node 1:
> > instance00/data/main_index_shard1_replica1
> > instance01/data/main_index_shard2_replica1
> > instance02/data/main_index_shard3_replica1
> > instance03/data/main_index_shard4_replica1
> >
> > Server node 2:
> > instance00/data/main_index_shard5_replica1
> > instance01/data/main_index_shard6_replica1
> > instance02/data/main_index_shard7_replica1
> > instance03/data/main_index_shard8_replica1
> >
> > However, while attempting to upgrade to 7.2.1, this numbering has
> changed:
> >
> > Solr 7.2.1 /var/lib/solr:
> >
> > Server node 1:
> > instance00/data/main_index_shard1_replica_n1
> > instance01/data/main_index_shard2_replica_n2
> > instance02/data/main_index_shard3_replica_n4
> > instance03/data/main_index_shard4_replica_n6
> >
> > Server node 2:
> > instance00/data/main_index_shard5_replica_n8
> > instance01/data/main_index_shard6_replica_n10
> > instance02/data/main_index_shard7_replica_n12
> > instance03/data/main_index_shard8_replica_n14
> >
> > This new numbering breaks my copy script, and furthermode, I'm
> worried
> > as to what happens when the numbering is different among target
> clusters.
> >
> > How can I switch this back to the old numbering scheme?
> >
> > Side note: is there a recommended way of doing this? Is the
> > backup/restore mechanism suitable for this? The ref guide is kind of
> terse
> > here.
> >
> > Thanks in advance,
> >
> > Ciao, Patrick
> >
> --
> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
> http://www.solrenterprisesearchserver.com
>
>
> --
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com

Re: Copying a SolrCloud collection to other hosts

[Jeff Wartes]

The backup/restore still requires setting up a shared filesystem on all your 
nodes though right? 

I've been using the fetchindex trick in my solrcloud_manager tool for ages now: 
https://github.com/whitepages/solrcloud_manager#cluster-commands
Some of the original features in that tool have been incorporated into Solr 
itself these days, but I still use clonecollection/copycollection regularly. 
(most recently with Solr 7.2)


On 3/27/18, 9:55 PM, "David Smiley"  wrote:

The backup/restore API is intended to address this.

https://builds.apache.org/job/Solr-reference-guide-master/javadoc/making-and-restoring-backups.html

Erick's advice is good (and I once drafted docs for the same scheme years
ago as well), but I consider it dated -- it's what people had to do before
the backup/restore API existed.  Internally, backup/restore is doing
similar stuff.  It's easy to give backup/restore a try; surely you have by
now?

~ David

On Tue, Mar 6, 2018 at 9:47 AM Patrick Schemitz  wrote:

> Hi List,
>
> so I'm running a bunch of SolrCloud clusters (each cluster is: 8 shards
> on 2 servers, with 4 instances per server, no replicas, i.e. 1 shard per
> instance).
>
> Building the index afresh takes 15+ hours, so when I have to deploy a new
> index, I build it once, on one cluster, and then copy (scp) over the
> data//index directories (shutting down the Solr instances
> first).
>
> I could get Solr 6.5.1 to number the shard/replica directories nicely via
> the createNodeSet and createNodeSet.shuffle options:
>
> Solr 6.5.1 /var/lib/solr:
>
> Server node 1:
> instance00/data/main_index_shard1_replica1
> instance01/data/main_index_shard2_replica1
> instance02/data/main_index_shard3_replica1
> instance03/data/main_index_shard4_replica1
>
> Server node 2:
> instance00/data/main_index_shard5_replica1
> instance01/data/main_index_shard6_replica1
> instance02/data/main_index_shard7_replica1
> instance03/data/main_index_shard8_replica1
>
> However, while attempting to upgrade to 7.2.1, this numbering has changed:
>
> Solr 7.2.1 /var/lib/solr:
>
> Server node 1:
> instance00/data/main_index_shard1_replica_n1
> instance01/data/main_index_shard2_replica_n2
> instance02/data/main_index_shard3_replica_n4
> instance03/data/main_index_shard4_replica_n6
>
> Server node 2:
> instance00/data/main_index_shard5_replica_n8
> instance01/data/main_index_shard6_replica_n10
> instance02/data/main_index_shard7_replica_n12
> instance03/data/main_index_shard8_replica_n14
>
> This new numbering breaks my copy script, and furthermode, I'm worried
> as to what happens when the numbering is different among target clusters.
>
> How can I switch this back to the old numbering scheme?
>
> Side note: is there a recommended way of doing this? Is the
> backup/restore mechanism suitable for this? The ref guide is kind of terse
> here.
>
> Thanks in advance,
>
> Ciao, Patrick
>
-- 
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com

Re: Copying a SolrCloud collection to other hosts

[David Smiley]

The backup/restore API is intended to address this.
https://builds.apache.org/job/Solr-reference-guide-master/javadoc/making-and-restoring-backups.html

Erick's advice is good (and I once drafted docs for the same scheme years
ago as well), but I consider it dated -- it's what people had to do before
the backup/restore API existed.  Internally, backup/restore is doing
similar stuff.  It's easy to give backup/restore a try; surely you have by
now?

~ David

On Tue, Mar 6, 2018 at 9:47 AM Patrick Schemitz  wrote:

> Hi List,
>
> so I'm running a bunch of SolrCloud clusters (each cluster is: 8 shards
> on 2 servers, with 4 instances per server, no replicas, i.e. 1 shard per
> instance).
>
> Building the index afresh takes 15+ hours, so when I have to deploy a new
> index, I build it once, on one cluster, and then copy (scp) over the
> data//index directories (shutting down the Solr instances
> first).
>
> I could get Solr 6.5.1 to number the shard/replica directories nicely via
> the createNodeSet and createNodeSet.shuffle options:
>
> Solr 6.5.1 /var/lib/solr:
>
> Server node 1:
> instance00/data/main_index_shard1_replica1
> instance01/data/main_index_shard2_replica1
> instance02/data/main_index_shard3_replica1
> instance03/data/main_index_shard4_replica1
>
> Server node 2:
> instance00/data/main_index_shard5_replica1
> instance01/data/main_index_shard6_replica1
> instance02/data/main_index_shard7_replica1
> instance03/data/main_index_shard8_replica1
>
> However, while attempting to upgrade to 7.2.1, this numbering has changed:
>
> Solr 7.2.1 /var/lib/solr:
>
> Server node 1:
> instance00/data/main_index_shard1_replica_n1
> instance01/data/main_index_shard2_replica_n2
> instance02/data/main_index_shard3_replica_n4
> instance03/data/main_index_shard4_replica_n6
>
> Server node 2:
> instance00/data/main_index_shard5_replica_n8
> instance01/data/main_index_shard6_replica_n10
> instance02/data/main_index_shard7_replica_n12
> instance03/data/main_index_shard8_replica_n14
>
> This new numbering breaks my copy script, and furthermode, I'm worried
> as to what happens when the numbering is different among target clusters.
>
> How can I switch this back to the old numbering scheme?
>
> Side note: is there a recommended way of doing this? Is the
> backup/restore mechanism suitable for this? The ref guide is kind of terse
> here.
>
> Thanks in advance,
>
> Ciao, Patrick
>
-- 
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com

Re: Copying a SolrCloud collection to other hosts

[Erick Erickson]

yeah, it's on a core-by-core basis. Which also makes getting it
propagated to all replicas something you have to be sure happens...

Glad it's working for you!
Erick

On Thu, Mar 15, 2018 at 1:54 AM, Patrick Schemitz  wrote:
> Hi Erick,
>
> thanks a lot, that solved our problem nicely.
>
> (It took us a try or two to notice that this will not copy the entire
> collection but only the shard on the source instance, and we need to do
> this for all instances explicitly. But hey, we had to do the same for
> the old approch of scp'ing the data directories.)
>
> Ciao, Patrick
>
> On Tue, Mar 06, 2018 at 07:18:15AM -0800, Erick Erickson wrote:
>> this is part of the "different replica types" capability, there are
>> NRT (the only type available prior to 7x), PULL and TLOG which would
>> have different names. I don't know of any way to switch it off.
>>
>> As far as moving the data, here's a little known trick: Use the
>> replication API to issue a fetchindexk, see:
>> https://lucene.apache.org/solr/guide/6_6/index-replication.html As
>> long as the target cluster can "see" the source cluster via http, this
>> should work. This is entirely outside SolrCloud and ZooKeeper is not
>> involved. This would even work with, say, one side being stand-alone
>> and the other being SolrCloud (not that you want to do that, just
>> illustrating it's not part of SolrCloud)...
>>
>> So you'd specify something like:
>> http://target_node:port/solr/core_name/replication?command=fetchindex=http://source_node:port/solr/core_name
>>
>> "core_name" in these cases is what appears in the "cores" dropdown on
>> the admin UI page. You do not have to shut Solr down at all on either
>> end to use this, although last I knew the target node would not serve
>> queries while this was happening.
>>
>> An alternative is to not hard-code the names in your copy script,
>> rather look at the information in ZooKeeper for your source and target
>> information, you could do this by using the CLUSTERSTATUS collections
>> API call.
>>
>> Best,
>> Erick
>>
>> On Tue, Mar 6, 2018 at 6:47 AM, Patrick Schemitz  wrote:
>> > Hi List,
>> >
>> > so I'm running a bunch of SolrCloud clusters (each cluster is: 8 shards
>> > on 2 servers, with 4 instances per server, no replicas, i.e. 1 shard per
>> > instance).
>> >
>> > Building the index afresh takes 15+ hours, so when I have to deploy a new
>> > index, I build it once, on one cluster, and then copy (scp) over the
>> > data//index directories (shutting down the Solr instances 
>> > first).
>> >
>> > I could get Solr 6.5.1 to number the shard/replica directories nicely via
>> > the createNodeSet and createNodeSet.shuffle options:
>> >
>> > Solr 6.5.1 /var/lib/solr:
>> >
>> > Server node 1:
>> > instance00/data/main_index_shard1_replica1
>> > instance01/data/main_index_shard2_replica1
>> > instance02/data/main_index_shard3_replica1
>> > instance03/data/main_index_shard4_replica1
>> >
>> > Server node 2:
>> > instance00/data/main_index_shard5_replica1
>> > instance01/data/main_index_shard6_replica1
>> > instance02/data/main_index_shard7_replica1
>> > instance03/data/main_index_shard8_replica1
>> >
>> > However, while attempting to upgrade to 7.2.1, this numbering has changed:
>> >
>> > Solr 7.2.1 /var/lib/solr:
>> >
>> > Server node 1:
>> > instance00/data/main_index_shard1_replica_n1
>> > instance01/data/main_index_shard2_replica_n2
>> > instance02/data/main_index_shard3_replica_n4
>> > instance03/data/main_index_shard4_replica_n6
>> >
>> > Server node 2:
>> > instance00/data/main_index_shard5_replica_n8
>> > instance01/data/main_index_shard6_replica_n10
>> > instance02/data/main_index_shard7_replica_n12
>> > instance03/data/main_index_shard8_replica_n14
>> >
>> > This new numbering breaks my copy script, and furthermode, I'm worried
>> > as to what happens when the numbering is different among target clusters.
>> >
>> > How can I switch this back to the old numbering scheme?
>> >
>> > Side note: is there a recommended way of doing this? Is the
>> > backup/restore mechanism suitable for this? The ref guide is kind of terse
>> > here.
>> >
>> > Thanks in advance,
>> >
>> > Ciao, Patrick

Re: Copying a SolrCloud collection to other hosts

[Patrick Schemitz]

Hi Erick,

thanks a lot, that solved our problem nicely.

(It took us a try or two to notice that this will not copy the entire
collection but only the shard on the source instance, and we need to do
this for all instances explicitly. But hey, we had to do the same for
the old approch of scp'ing the data directories.)

Ciao, Patrick

On Tue, Mar 06, 2018 at 07:18:15AM -0800, Erick Erickson wrote:
> this is part of the "different replica types" capability, there are
> NRT (the only type available prior to 7x), PULL and TLOG which would
> have different names. I don't know of any way to switch it off.
> 
> As far as moving the data, here's a little known trick: Use the
> replication API to issue a fetchindexk, see:
> https://lucene.apache.org/solr/guide/6_6/index-replication.html As
> long as the target cluster can "see" the source cluster via http, this
> should work. This is entirely outside SolrCloud and ZooKeeper is not
> involved. This would even work with, say, one side being stand-alone
> and the other being SolrCloud (not that you want to do that, just
> illustrating it's not part of SolrCloud)...
> 
> So you'd specify something like:
> http://target_node:port/solr/core_name/replication?command=fetchindex=http://source_node:port/solr/core_name
> 
> "core_name" in these cases is what appears in the "cores" dropdown on
> the admin UI page. You do not have to shut Solr down at all on either
> end to use this, although last I knew the target node would not serve
> queries while this was happening.
> 
> An alternative is to not hard-code the names in your copy script,
> rather look at the information in ZooKeeper for your source and target
> information, you could do this by using the CLUSTERSTATUS collections
> API call.
> 
> Best,
> Erick
> 
> On Tue, Mar 6, 2018 at 6:47 AM, Patrick Schemitz  wrote:
> > Hi List,
> >
> > so I'm running a bunch of SolrCloud clusters (each cluster is: 8 shards
> > on 2 servers, with 4 instances per server, no replicas, i.e. 1 shard per
> > instance).
> >
> > Building the index afresh takes 15+ hours, so when I have to deploy a new
> > index, I build it once, on one cluster, and then copy (scp) over the
> > data//index directories (shutting down the Solr instances 
> > first).
> >
> > I could get Solr 6.5.1 to number the shard/replica directories nicely via
> > the createNodeSet and createNodeSet.shuffle options:
> >
> > Solr 6.5.1 /var/lib/solr:
> >
> > Server node 1:
> > instance00/data/main_index_shard1_replica1
> > instance01/data/main_index_shard2_replica1
> > instance02/data/main_index_shard3_replica1
> > instance03/data/main_index_shard4_replica1
> >
> > Server node 2:
> > instance00/data/main_index_shard5_replica1
> > instance01/data/main_index_shard6_replica1
> > instance02/data/main_index_shard7_replica1
> > instance03/data/main_index_shard8_replica1
> >
> > However, while attempting to upgrade to 7.2.1, this numbering has changed:
> >
> > Solr 7.2.1 /var/lib/solr:
> >
> > Server node 1:
> > instance00/data/main_index_shard1_replica_n1
> > instance01/data/main_index_shard2_replica_n2
> > instance02/data/main_index_shard3_replica_n4
> > instance03/data/main_index_shard4_replica_n6
> >
> > Server node 2:
> > instance00/data/main_index_shard5_replica_n8
> > instance01/data/main_index_shard6_replica_n10
> > instance02/data/main_index_shard7_replica_n12
> > instance03/data/main_index_shard8_replica_n14
> >
> > This new numbering breaks my copy script, and furthermode, I'm worried
> > as to what happens when the numbering is different among target clusters.
> >
> > How can I switch this back to the old numbering scheme?
> >
> > Side note: is there a recommended way of doing this? Is the
> > backup/restore mechanism suitable for this? The ref guide is kind of terse
> > here.
> >
> > Thanks in advance,
> >
> > Ciao, Patrick

Re: Copying a SolrCloud collection to other hosts

[Erick Erickson]

this is part of the "different replica types" capability, there are
NRT (the only type available prior to 7x), PULL and TLOG which would
have different names. I don't know of any way to switch it off.

As far as moving the data, here's a little known trick: Use the
replication API to issue a fetchindexk, see:
https://lucene.apache.org/solr/guide/6_6/index-replication.html As
long as the target cluster can "see" the source cluster via http, this
should work. This is entirely outside SolrCloud and ZooKeeper is not
involved. This would even work with, say, one side being stand-alone
and the other being SolrCloud (not that you want to do that, just
illustrating it's not part of SolrCloud)...

So you'd specify something like:
http://target_node:port/solr/core_name/replication?command=fetchindex=http://source_node:port/solr/core_name

"core_name" in these cases is what appears in the "cores" dropdown on
the admin UI page. You do not have to shut Solr down at all on either
end to use this, although last I knew the target node would not serve
queries while this was happening.

An alternative is to not hard-code the names in your copy script,
rather look at the information in ZooKeeper for your source and target
information, you could do this by using the CLUSTERSTATUS collections
API call.

Best,
Erick

On Tue, Mar 6, 2018 at 6:47 AM, Patrick Schemitz  wrote:
> Hi List,
>
> so I'm running a bunch of SolrCloud clusters (each cluster is: 8 shards
> on 2 servers, with 4 instances per server, no replicas, i.e. 1 shard per
> instance).
>
> Building the index afresh takes 15+ hours, so when I have to deploy a new
> index, I build it once, on one cluster, and then copy (scp) over the
> data//index directories (shutting down the Solr instances first).
>
> I could get Solr 6.5.1 to number the shard/replica directories nicely via
> the createNodeSet and createNodeSet.shuffle options:
>
> Solr 6.5.1 /var/lib/solr:
>
> Server node 1:
> instance00/data/main_index_shard1_replica1
> instance01/data/main_index_shard2_replica1
> instance02/data/main_index_shard3_replica1
> instance03/data/main_index_shard4_replica1
>
> Server node 2:
> instance00/data/main_index_shard5_replica1
> instance01/data/main_index_shard6_replica1
> instance02/data/main_index_shard7_replica1
> instance03/data/main_index_shard8_replica1
>
> However, while attempting to upgrade to 7.2.1, this numbering has changed:
>
> Solr 7.2.1 /var/lib/solr:
>
> Server node 1:
> instance00/data/main_index_shard1_replica_n1
> instance01/data/main_index_shard2_replica_n2
> instance02/data/main_index_shard3_replica_n4
> instance03/data/main_index_shard4_replica_n6
>
> Server node 2:
> instance00/data/main_index_shard5_replica_n8
> instance01/data/main_index_shard6_replica_n10
> instance02/data/main_index_shard7_replica_n12
> instance03/data/main_index_shard8_replica_n14
>
> This new numbering breaks my copy script, and furthermode, I'm worried
> as to what happens when the numbering is different among target clusters.
>
> How can I switch this back to the old numbering scheme?
>
> Side note: is there a recommended way of doing this? Is the
> backup/restore mechanism suitable for this? The ref guide is kind of terse
> here.
>
> Thanks in advance,
>
> Ciao, Patrick