Re: Challenges with new Solrcloud Backup/Restore functionality

[Hrishikesh Gadre]

Hi Stephen,

Thanks for the update.

Regarding SOLR-9527 - I think we need a unit test for verifying
"createNodeSet" functionality. I will spend some time on it in next couple
of days.

Also regarding #2, i also found similar issue (doc count mismatch after
restore) while testing with a large collection (~50GB index size). I have
opened SOLR-9598 to track this. Please take a look and comment if you have
any insight.

-Hrishikesh

On Tue, Nov 8, 2016 at 12:54 PM, Stephen Weiss  wrote:

> Just wanted to note that we tested out the patch from SOLR-9527 and it
> worked perfectly for the balancing issue - thank you so much for that!
>
> As for issue #2, we've resorted to doing a hard commit, stopping all
> indexing against the index, and then taking the backup, and we have a
> reasonably good success rate with that.  The system is set up to
> automatically delete and retry the backup/restore process if the cores
> don't match, so that's allowed us to smooth over that problem and get this
> process out into production.   We've been using it for several weeks now
> without any major issue!
>
> We just looked because Solr 6.3 was out, and wanted to know if we could
> upgrade without patching again, but it appears this ticket hasn't gone
> anywhere yet.  I know one users' testing is probably not enough, but given
> that it seems the patch works just fine, are there any plans to merge it
> into release yet?
>
> --
> Steve
>
> On Tue, Oct 4, 2016 at 6:46 PM, Stephen Lewis > wrote:
> Hi All,
>
> I have been experiencing error#1 too with the current branch_6_2 build. I
> started noticing after I applied my patch to that branch<
> https://issues.apache.org/jira/browse/SOLR-9527> (on issue #2), but it
> appears to occur without the patch as well. I haven't seen this issue with
> solr 6.1.0 despite extensive testing. I haven't confirmed if this occurs on
> the official 6.2.0 release build. I will try to confirm and gather more
> data soon.
>
> As with Stephen Weiss, I also am not seeing any errors logged in the index
> after backup and the task is marked as succeeded. However, after each
> backup which is missing a large amount of data, the restore command fails,
> in the sense that the collection is created, but the initialized cores are
> blank and the logs contain errors about "incomplete segments". I will try
> to research further and get back with more data soon.
>
>
>
> On Mon, Sep 26, 2016 at 11:26 AM, Hrishikesh Gadre  > wrote:
> Hi Stephen,
>
> regarding #1, can you verify following steps during backup/restore?
>
> - Before backup command, make sure to run a "hard" commit on the original
> collection. The backup operation will capture only hard committed data.
> - After restore command, check the Solr web UI to verify that all replicas
> of the new (or restored) collection are in the "active" state. During my
> testing, I found that when one or more replicas are in "recovery" state,
> the doc count of the restored collection doesn't match the doc count of the
> original collection. But after the recovery is complete, the doc counts
> match. I will file a JIRA to fix this issue.
>
> Thanks
> Hrishikesh
>
> On Mon, Sep 26, 2016 at 9:34 AM, Stephen Weiss  > wrote:
>
> > #2 - that's great news.  I'll try to patch it in and test it out.
> >
> > #1 - In all cases, the backup and restore both appear successful.  There
> > are no failure messages for any of the shards, no warnings, etc - I
> didn't
> > even realize at first that data was missing until I noticed differences
> in
> > some of the query results when we were testing.  Either manual restore of
> > the data or using the restore API (with all data on one node), we see the
> > same, so I think it's more a problem in the backup process than the
> restore
> > process.
> >
> > If there's any kind of debugging output we can provide that can help
> solve
> > this, let me know.
> >
> > --
> > Steve
> >
> > On Sun, Sep 25, 2016 at 7:17 PM, Hrishikesh Gadre  >
> > wrote:
> >
> >> Hi Steve,
> >>
> >> Regarding the 2nd issue, a JIRA is already created and patch is uploaded
> >> (SOLR-9527). Can someone review and commit the patch?
> >>
> >> Regarding 1st issue, does backup command succeeds? Also do you see any
> >> warning/error log messages? How about the restore command?
> >>
> >> Thanks
> >> Hrishikesh
> >>
> >>
> >>
> >> On Sat, Sep 24, 2016 at 12:14 PM, Stephen Weiss  >
> >> wrote:
> >>
> >>> Hi everyone,
> >>>
> >>> We're very excited about SolrCloud's new backup / restore collection
> >>> APIs, which should introduce some major new efficiencies into our
> indexing
> >>> workflow.  Unfortunately, we've run into some snags with it that are
> >>> preventing us from moving into production.  I was hoping 

Re: Challenges with new Solrcloud Backup/Restore functionality

[Stephen Weiss]

Just wanted to note that we tested out the patch from SOLR-9527 and it worked 
perfectly for the balancing issue - thank you so much for that!

As for issue #2, we've resorted to doing a hard commit, stopping all indexing 
against the index, and then taking the backup, and we have a reasonably good 
success rate with that.  The system is set up to automatically delete and retry 
the backup/restore process if the cores don't match, so that's allowed us to 
smooth over that problem and get this process out into production.   We've been 
using it for several weeks now without any major issue!

We just looked because Solr 6.3 was out, and wanted to know if we could upgrade 
without patching again, but it appears this ticket hasn't gone anywhere yet.  I 
know one users' testing is probably not enough, but given that it seems the 
patch works just fine, are there any plans to merge it into release yet?

--
Steve

On Tue, Oct 4, 2016 at 6:46 PM, Stephen Lewis 
> wrote:
Hi All,

I have been experiencing error#1 too with the current branch_6_2 build. I 
started noticing after I applied my patch to that 
branch (on issue #2), but it 
appears to occur without the patch as well. I haven't seen this issue with solr 
6.1.0 despite extensive testing. I haven't confirmed if this occurs on the 
official 6.2.0 release build. I will try to confirm and gather more data soon.

As with Stephen Weiss, I also am not seeing any errors logged in the index 
after backup and the task is marked as succeeded. However, after each backup 
which is missing a large amount of data, the restore command fails, in the 
sense that the collection is created, but the initialized cores are blank and 
the logs contain errors about "incomplete segments". I will try to research 
further and get back with more data soon.



On Mon, Sep 26, 2016 at 11:26 AM, Hrishikesh Gadre 
> wrote:
Hi Stephen,

regarding #1, can you verify following steps during backup/restore?

- Before backup command, make sure to run a "hard" commit on the original
collection. The backup operation will capture only hard committed data.
- After restore command, check the Solr web UI to verify that all replicas
of the new (or restored) collection are in the "active" state. During my
testing, I found that when one or more replicas are in "recovery" state,
the doc count of the restored collection doesn't match the doc count of the
original collection. But after the recovery is complete, the doc counts
match. I will file a JIRA to fix this issue.

Thanks
Hrishikesh

On Mon, Sep 26, 2016 at 9:34 AM, Stephen Weiss 
> wrote:

> #2 - that's great news.  I'll try to patch it in and test it out.
>
> #1 - In all cases, the backup and restore both appear successful.  There
> are no failure messages for any of the shards, no warnings, etc - I didn't
> even realize at first that data was missing until I noticed differences in
> some of the query results when we were testing.  Either manual restore of
> the data or using the restore API (with all data on one node), we see the
> same, so I think it's more a problem in the backup process than the restore
> process.
>
> If there's any kind of debugging output we can provide that can help solve
> this, let me know.
>
> --
> Steve
>
> On Sun, Sep 25, 2016 at 7:17 PM, Hrishikesh Gadre 
> >
> wrote:
>
>> Hi Steve,
>>
>> Regarding the 2nd issue, a JIRA is already created and patch is uploaded
>> (SOLR-9527). Can someone review and commit the patch?
>>
>> Regarding 1st issue, does backup command succeeds? Also do you see any
>> warning/error log messages? How about the restore command?
>>
>> Thanks
>> Hrishikesh
>>
>>
>>
>> On Sat, Sep 24, 2016 at 12:14 PM, Stephen Weiss 
>> >
>> wrote:
>>
>>> Hi everyone,
>>>
>>> We're very excited about SolrCloud's new backup / restore collection
>>> APIs, which should introduce some major new efficiencies into our indexing
>>> workflow.  Unfortunately, we've run into some snags with it that are
>>> preventing us from moving into production.  I was hoping someone on the
>>> list could help.
>>>
>>> 1) Data inconsistencies
>>>
>>> There seems to be a problem getting all the data consistently.
>>> Sometimes, the backup will contain all of the data in the collection, and
>>> sometimes, large portions of the collection (as much as 40%) will be
>>> missing.  We haven't quite figured out what might cause this yet, although
>>> one thing I've noticed is the chances of success are greater when we are
>>> only backing up one collection at a time.  Unfortunately, for our workflow,
>>> it will be difficult to make that work, and there still doesn't seem to be
>>> a guarantee of success either way.
>>>
>>> 2) Shards are not distributed
>>>

Re: Challenges with new Solrcloud Backup/Restore functionality

[Stephen Lewis]

Hi All,

I have been experiencing error#1 too with the current branch_6_2 build. I
started noticing after I applied my patch to that branch
 (on issue #2), but it
appears to occur without the patch as well. I haven't seen this issue with
solr 6.1.0 despite extensive testing. I haven't confirmed if this occurs on
the official 6.2.0 release build. I will try to confirm and gather more
data soon.

As with Stephen Weiss, I also am not seeing any errors logged in the index
after backup and the task is marked as succeeded. However, after each
backup which is missing a large amount of data, the restore command fails,
in the sense that the collection is created, but the initialized cores are
blank and the logs contain errors about "incomplete segments". I will try
to research further and get back with more data soon.



On Mon, Sep 26, 2016 at 11:26 AM, Hrishikesh Gadre 
wrote:

> Hi Stephen,
>
> regarding #1, can you verify following steps during backup/restore?
>
> - Before backup command, make sure to run a "hard" commit on the original
> collection. The backup operation will capture only hard committed data.
> - After restore command, check the Solr web UI to verify that all replicas
> of the new (or restored) collection are in the "active" state. During my
> testing, I found that when one or more replicas are in "recovery" state,
> the doc count of the restored collection doesn't match the doc count of the
> original collection. But after the recovery is complete, the doc counts
> match. I will file a JIRA to fix this issue.
>
> Thanks
> Hrishikesh
>
> On Mon, Sep 26, 2016 at 9:34 AM, Stephen Weiss 
> wrote:
>
> > #2 - that's great news.  I'll try to patch it in and test it out.
> >
> > #1 - In all cases, the backup and restore both appear successful.  There
> > are no failure messages for any of the shards, no warnings, etc - I
> didn't
> > even realize at first that data was missing until I noticed differences
> in
> > some of the query results when we were testing.  Either manual restore of
> > the data or using the restore API (with all data on one node), we see the
> > same, so I think it's more a problem in the backup process than the
> restore
> > process.
> >
> > If there's any kind of debugging output we can provide that can help
> solve
> > this, let me know.
> >
> > --
> > Steve
> >
> > On Sun, Sep 25, 2016 at 7:17 PM, Hrishikesh Gadre 
> > wrote:
> >
> >> Hi Steve,
> >>
> >> Regarding the 2nd issue, a JIRA is already created and patch is uploaded
> >> (SOLR-9527). Can someone review and commit the patch?
> >>
> >> Regarding 1st issue, does backup command succeeds? Also do you see any
> >> warning/error log messages? How about the restore command?
> >>
> >> Thanks
> >> Hrishikesh
> >>
> >>
> >>
> >> On Sat, Sep 24, 2016 at 12:14 PM, Stephen Weiss 
> >> wrote:
> >>
> >>> Hi everyone,
> >>>
> >>> We're very excited about SolrCloud's new backup / restore collection
> >>> APIs, which should introduce some major new efficiencies into our
> indexing
> >>> workflow.  Unfortunately, we've run into some snags with it that are
> >>> preventing us from moving into production.  I was hoping someone on the
> >>> list could help.
> >>>
> >>> 1) Data inconsistencies
> >>>
> >>> There seems to be a problem getting all the data consistently.
> >>> Sometimes, the backup will contain all of the data in the collection,
> and
> >>> sometimes, large portions of the collection (as much as 40%) will be
> >>> missing.  We haven't quite figured out what might cause this yet,
> although
> >>> one thing I've noticed is the chances of success are greater when we
> are
> >>> only backing up one collection at a time.  Unfortunately, for our
> workflow,
> >>> it will be difficult to make that work, and there still doesn't seem
> to be
> >>> a guarantee of success either way.
> >>>
> >>> 2) Shards are not distributed
> >>>
> >>> To make matters worse, for some reason, any type of restore operation
> >>> always seems to put all shards of the collection on the same node.
> We've
> >>> tried setting maxShardsPerNode to 1 in the restore command, but this
> has no
> >>> effect.  We are seeing the same behavior on both 6.1 and 6.2.1.  No
> matter
> >>> what we do, all the shards always go to the same node - and it's not
> even
> >>> the node that we execute the restore request on, but oddly enough, a
> >>> totally different node, and always the same one (the 4th one).  It
> should
> >>> be noted that all nodes of our 8 node cloud are up and totally
> functional
> >>> when this happens.
> >>>
> >>> To work around this, we wrote up a quick script to create an empty
> >>> collection, which always distributes itself across the cloud quite well
> >>> (another indication that there's nothing wrong with the nodes
> themselves),
> >>> and then we rsync the individual shards' data into the empty shards and
> 

Re: Challenges with new Solrcloud Backup/Restore functionality

[Hrishikesh Gadre]

Hi Stephen,

regarding #1, can you verify following steps during backup/restore?

- Before backup command, make sure to run a "hard" commit on the original
collection. The backup operation will capture only hard committed data.
- After restore command, check the Solr web UI to verify that all replicas
of the new (or restored) collection are in the "active" state. During my
testing, I found that when one or more replicas are in "recovery" state,
the doc count of the restored collection doesn't match the doc count of the
original collection. But after the recovery is complete, the doc counts
match. I will file a JIRA to fix this issue.

Thanks
Hrishikesh

On Mon, Sep 26, 2016 at 9:34 AM, Stephen Weiss  wrote:

> #2 - that's great news.  I'll try to patch it in and test it out.
>
> #1 - In all cases, the backup and restore both appear successful.  There
> are no failure messages for any of the shards, no warnings, etc - I didn't
> even realize at first that data was missing until I noticed differences in
> some of the query results when we were testing.  Either manual restore of
> the data or using the restore API (with all data on one node), we see the
> same, so I think it's more a problem in the backup process than the restore
> process.
>
> If there's any kind of debugging output we can provide that can help solve
> this, let me know.
>
> --
> Steve
>
> On Sun, Sep 25, 2016 at 7:17 PM, Hrishikesh Gadre 
> wrote:
>
>> Hi Steve,
>>
>> Regarding the 2nd issue, a JIRA is already created and patch is uploaded
>> (SOLR-9527). Can someone review and commit the patch?
>>
>> Regarding 1st issue, does backup command succeeds? Also do you see any
>> warning/error log messages? How about the restore command?
>>
>> Thanks
>> Hrishikesh
>>
>>
>>
>> On Sat, Sep 24, 2016 at 12:14 PM, Stephen Weiss 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> We're very excited about SolrCloud's new backup / restore collection
>>> APIs, which should introduce some major new efficiencies into our indexing
>>> workflow.  Unfortunately, we've run into some snags with it that are
>>> preventing us from moving into production.  I was hoping someone on the
>>> list could help.
>>>
>>> 1) Data inconsistencies
>>>
>>> There seems to be a problem getting all the data consistently.
>>> Sometimes, the backup will contain all of the data in the collection, and
>>> sometimes, large portions of the collection (as much as 40%) will be
>>> missing.  We haven't quite figured out what might cause this yet, although
>>> one thing I've noticed is the chances of success are greater when we are
>>> only backing up one collection at a time.  Unfortunately, for our workflow,
>>> it will be difficult to make that work, and there still doesn't seem to be
>>> a guarantee of success either way.
>>>
>>> 2) Shards are not distributed
>>>
>>> To make matters worse, for some reason, any type of restore operation
>>> always seems to put all shards of the collection on the same node.  We've
>>> tried setting maxShardsPerNode to 1 in the restore command, but this has no
>>> effect.  We are seeing the same behavior on both 6.1 and 6.2.1.  No matter
>>> what we do, all the shards always go to the same node - and it's not even
>>> the node that we execute the restore request on, but oddly enough, a
>>> totally different node, and always the same one (the 4th one).  It should
>>> be noted that all nodes of our 8 node cloud are up and totally functional
>>> when this happens.
>>>
>>> To work around this, we wrote up a quick script to create an empty
>>> collection, which always distributes itself across the cloud quite well
>>> (another indication that there's nothing wrong with the nodes themselves),
>>> and then we rsync the individual shards' data into the empty shards and
>>> reload the collection.  This works fine, however, because of the data
>>> inconsistencies mentioned above, we can't really move forward anyway.
>>>
>>>
>>> Problem #2, we have a reasonable workaround for, but problem #1 we do
>>> not.  If anyone has any thoughts about either of these problems, I would be
>>> very grateful.  Thanks!
>>>
>>> --
>>> Steve
>>>
>>> 
>>>
>>> WGSN is a global foresight business. Our experts provide deep insight
>>> and analysis of consumer, fashion and design trends. We inspire our clients
>>> to plan and trade their range with unparalleled confidence and accuracy.
>>> Together, we Create Tomorrow.
>>>
>>> WGSN is part of WGSN Limited, comprising of
>>> market-leading products including WGSN.com, WGSN
>>> Lifestyle & Interiors, WGSN
>>> INstock, WGSN StyleTrial<
>>> http://www.wgsn.com/en/styletrial/> and WGSN Mindset<
>>> http://www.wgsn.com/en/services/consultancy/>, our bespoke consultancy
>>> services.
>>>
>>> The information in or attached to this email is 

Re: Challenges with new Solrcloud Backup/Restore functionality

[Stephen Weiss]

#2 - that's great news.  I'll try to patch it in and test it out.

#1 - In all cases, the backup and restore both appear successful.  There are no 
failure messages for any of the shards, no warnings, etc - I didn't even 
realize at first that data was missing until I noticed differences in some of 
the query results when we were testing.  Either manual restore of the data or 
using the restore API (with all data on one node), we see the same, so I think 
it's more a problem in the backup process than the restore process.

If there's any kind of debugging output we can provide that can help solve 
this, let me know.

--
Steve

On Sun, Sep 25, 2016 at 7:17 PM, Hrishikesh Gadre 
> wrote:
Hi Steve,

Regarding the 2nd issue, a JIRA is already created and patch is uploaded 
(SOLR-9527). Can someone review and commit the patch?

Regarding 1st issue, does backup command succeeds? Also do you see any 
warning/error log messages? How about the restore command?

Thanks
Hrishikesh



On Sat, Sep 24, 2016 at 12:14 PM, Stephen Weiss 
> wrote:
Hi everyone,

We're very excited about SolrCloud's new backup / restore collection APIs, 
which should introduce some major new efficiencies into our indexing workflow.  
Unfortunately, we've run into some snags with it that are preventing us from 
moving into production.  I was hoping someone on the list could help.

1) Data inconsistencies

There seems to be a problem getting all the data consistently.  Sometimes, the 
backup will contain all of the data in the collection, and sometimes, large 
portions of the collection (as much as 40%) will be missing.  We haven't quite 
figured out what might cause this yet, although one thing I've noticed is the 
chances of success are greater when we are only backing up one collection at a 
time.  Unfortunately, for our workflow, it will be difficult to make that work, 
and there still doesn't seem to be a guarantee of success either way.

2) Shards are not distributed

To make matters worse, for some reason, any type of restore operation always 
seems to put all shards of the collection on the same node.  We've tried 
setting maxShardsPerNode to 1 in the restore command, but this has no effect.  
We are seeing the same behavior on both 6.1 and 6.2.1.  No matter what we do, 
all the shards always go to the same node - and it's not even the node that we 
execute the restore request on, but oddly enough, a totally different node, and 
always the same one (the 4th one).  It should be noted that all nodes of our 8 
node cloud are up and totally functional when this happens.

To work around this, we wrote up a quick script to create an empty collection, 
which always distributes itself across the cloud quite well (another indication 
that there's nothing wrong with the nodes themselves), and then we rsync the 
individual shards' data into the empty shards and reload the collection.  This 
works fine, however, because of the data inconsistencies mentioned above, we 
can't really move forward anyway.


Problem #2, we have a reasonable workaround for, but problem #1 we do not.  If 
anyone has any thoughts about either of these problems, I would be very 
grateful.  Thanks!

--
Steve



WGSN is a global foresight business. Our experts provide deep insight and 
analysis of consumer, fashion and design trends. We inspire our clients to plan 
and trade their range with unparalleled confidence and accuracy. Together, we 
Create Tomorrow.

WGSN is part of WGSN Limited, comprising of 
market-leading products including WGSN.com, WGSN Lifestyle 
& Interiors, WGSN 
INstock, WGSN 
StyleTrial and WGSN 
Mindset, our bespoke consultancy 
services.

The information in or attached to this email is confidential and may be legally 
privileged. If you are not the intended recipient of this message, any use, 
disclosure, copying, distribution or any action taken in reliance on it is 
prohibited and may be unlawful. If you have received this message in error, 
please notify the sender immediately by return email and delete this message 
and any copies from your computer and network. WGSN does not warrant that this 
email and any attachments are free from viruses and accepts no liability for 
any loss resulting from infected email transmissions.

WGSN reserves the right to monitor all email through its networks. Any views 
expressed may be those of the originator and not necessarily of WGSN. WGSN is 
powered by Ascential plc, which transforms knowledge 
businesses to deliver exceptional performance.

Please be advised all phone calls may be recorded for training and quality 
purposes and by accepting and/or making calls 

Re: Challenges with new Solrcloud Backup/Restore functionality

[Hrishikesh Gadre]

Hi Steve,

Regarding the 2nd issue, a JIRA is already created and patch is uploaded
(SOLR-9527). Can someone review and commit the patch?

Regarding 1st issue, does backup command succeeds? Also do you see any
warning/error log messages? How about the restore command?

Thanks
Hrishikesh



On Sat, Sep 24, 2016 at 12:14 PM, Stephen Weiss 
wrote:

> Hi everyone,
>
> We're very excited about SolrCloud's new backup / restore collection APIs,
> which should introduce some major new efficiencies into our indexing
> workflow.  Unfortunately, we've run into some snags with it that are
> preventing us from moving into production.  I was hoping someone on the
> list could help.
>
> 1) Data inconsistencies
>
> There seems to be a problem getting all the data consistently.  Sometimes,
> the backup will contain all of the data in the collection, and sometimes,
> large portions of the collection (as much as 40%) will be missing.  We
> haven't quite figured out what might cause this yet, although one thing
> I've noticed is the chances of success are greater when we are only backing
> up one collection at a time.  Unfortunately, for our workflow, it will be
> difficult to make that work, and there still doesn't seem to be a guarantee
> of success either way.
>
> 2) Shards are not distributed
>
> To make matters worse, for some reason, any type of restore operation
> always seems to put all shards of the collection on the same node.  We've
> tried setting maxShardsPerNode to 1 in the restore command, but this has no
> effect.  We are seeing the same behavior on both 6.1 and 6.2.1.  No matter
> what we do, all the shards always go to the same node - and it's not even
> the node that we execute the restore request on, but oddly enough, a
> totally different node, and always the same one (the 4th one).  It should
> be noted that all nodes of our 8 node cloud are up and totally functional
> when this happens.
>
> To work around this, we wrote up a quick script to create an empty
> collection, which always distributes itself across the cloud quite well
> (another indication that there's nothing wrong with the nodes themselves),
> and then we rsync the individual shards' data into the empty shards and
> reload the collection.  This works fine, however, because of the data
> inconsistencies mentioned above, we can't really move forward anyway.
>
>
> Problem #2, we have a reasonable workaround for, but problem #1 we do
> not.  If anyone has any thoughts about either of these problems, I would be
> very grateful.  Thanks!
>
> --
> Steve
>
> 
>
> WGSN is a global foresight business. Our experts provide deep insight and
> analysis of consumer, fashion and design trends. We inspire our clients to
> plan and trade their range with unparalleled confidence and accuracy.
> Together, we Create Tomorrow.
>
> WGSN is part of WGSN Limited, comprising of
> market-leading products including WGSN.com, WGSN
> Lifestyle & Interiors, WGSN
> INstock, WGSN StyleTrial com/en/styletrial/> and WGSN Mindset en/services/consultancy/>, our bespoke consultancy services.
>
> The information in or attached to this email is confidential and may be
> legally privileged. If you are not the intended recipient of this message,
> any use, disclosure, copying, distribution or any action taken in reliance
> on it is prohibited and may be unlawful. If you have received this message
> in error, please notify the sender immediately by return email and delete
> this message and any copies from your computer and network. WGSN does not
> warrant that this email and any attachments are free from viruses and
> accepts no liability for any loss resulting from infected email
> transmissions.
>
> WGSN reserves the right to monitor all email through its networks. Any
> views expressed may be those of the originator and not necessarily of WGSN.
> WGSN is powered by Ascential plc, which
> transforms knowledge businesses to deliver exceptional performance.
>
> Please be advised all phone calls may be recorded for training and quality
> purposes and by accepting and/or making calls from and/or to us you
> acknowledge and agree to calls being recorded.
>
> WGSN Limited, Company number 4858491
>
> registered address:
>
> Ascential plc, The Prow, 1 Wilder Walk, London W1B 5AP
>
> WGSN Inc., tax ID 04-3851246, registered office c/o National Registered
> Agents, Inc., 160 Greentree Drive, Suite 101, Dover DE 19904, United States
>
> 4C Serviços de Informação Ltda., CNPJ/MF (Taxpayer's Register):
> 15.536.968/0001-04, Address: Avenida Cidade Jardim, 377, 7˚ andar CEP
> 01453-000, Itaim Bibi, São Paulo
>
> 4C Business Information Consulting (Shanghai) Co., Ltd, 富新商务信息咨询（上海）有限公司,
> registered address Unit 4810/4811, 48/F Tower 1, Grand Gateway, 1 Hong 

Challenges with new Solrcloud Backup/Restore functionality

[Stephen Weiss]

Hi everyone,

We're very excited about SolrCloud's new backup / restore collection APIs, 
which should introduce some major new efficiencies into our indexing workflow.  
Unfortunately, we've run into some snags with it that are preventing us from 
moving into production.  I was hoping someone on the list could help.

1) Data inconsistencies

There seems to be a problem getting all the data consistently.  Sometimes, the 
backup will contain all of the data in the collection, and sometimes, large 
portions of the collection (as much as 40%) will be missing.  We haven't quite 
figured out what might cause this yet, although one thing I've noticed is the 
chances of success are greater when we are only backing up one collection at a 
time.  Unfortunately, for our workflow, it will be difficult to make that work, 
and there still doesn't seem to be a guarantee of success either way.

2) Shards are not distributed

To make matters worse, for some reason, any type of restore operation always 
seems to put all shards of the collection on the same node.  We've tried 
setting maxShardsPerNode to 1 in the restore command, but this has no effect.  
We are seeing the same behavior on both 6.1 and 6.2.1.  No matter what we do, 
all the shards always go to the same node - and it's not even the node that we 
execute the restore request on, but oddly enough, a totally different node, and 
always the same one (the 4th one).  It should be noted that all nodes of our 8 
node cloud are up and totally functional when this happens.

To work around this, we wrote up a quick script to create an empty collection, 
which always distributes itself across the cloud quite well (another indication 
that there's nothing wrong with the nodes themselves), and then we rsync the 
individual shards' data into the empty shards and reload the collection.  This 
works fine, however, because of the data inconsistencies mentioned above, we 
can't really move forward anyway.


Problem #2, we have a reasonable workaround for, but problem #1 we do not.  If 
anyone has any thoughts about either of these problems, I would be very 
grateful.  Thanks!

--
Steve



WGSN is a global foresight business. Our experts provide deep insight and 
analysis of consumer, fashion and design trends. We inspire our clients to plan 
and trade their range with unparalleled confidence and accuracy. Together, we 
Create Tomorrow.

WGSN is part of WGSN Limited, comprising of 
market-leading products including WGSN.com, WGSN Lifestyle 
& Interiors, WGSN 
INstock, WGSN 
StyleTrial and WGSN 
Mindset, our bespoke consultancy 
services.

The information in or attached to this email is confidential and may be legally 
privileged. If you are not the intended recipient of this message, any use, 
disclosure, copying, distribution or any action taken in reliance on it is 
prohibited and may be unlawful. If you have received this message in error, 
please notify the sender immediately by return email and delete this message 
and any copies from your computer and network. WGSN does not warrant that this 
email and any attachments are free from viruses and accepts no liability for 
any loss resulting from infected email transmissions.

WGSN reserves the right to monitor all email through its networks. Any views 
expressed may be those of the originator and not necessarily of WGSN. WGSN is 
powered by Ascential plc, which transforms knowledge 
businesses to deliver exceptional performance.

Please be advised all phone calls may be recorded for training and quality 
purposes and by accepting and/or making calls from and/or to us you acknowledge 
and agree to calls being recorded.

WGSN Limited, Company number 4858491

registered address:

Ascential plc, The Prow, 1 Wilder Walk, London W1B 5AP

WGSN Inc., tax ID 04-3851246, registered office c/o National Registered Agents, 
Inc., 160 Greentree Drive, Suite 101, Dover DE 19904, United States

4C Serviços de Informação Ltda., CNPJ/MF (Taxpayer's Register): 
15.536.968/0001-04, Address: Avenida Cidade Jardim, 377, 7˚ andar CEP 
01453-000, Itaim Bibi, São Paulo

4C Business Information Consulting (Shanghai) Co., Ltd, 富新商务信息咨询（上海）有限公司, 
registered address Unit 4810/4811, 48/F Tower 1, Grand Gateway, 1 Hong Qiao 
Road, Xuhui District, Shanghai

Re: SolrCloud backup/restore

[Zisis Tachtsidis]

Thank you both for the clarification and proposals!

This solrcloud_manager looks very promising. I'll try it out, the shared
filesystem requirement is no issue for me. 



--
View this message in context: 
http://lucene.472066.n3.nabble.com/SolrCloud-backup-restore-tp4267954p4268197.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrCloud backup/restore

[Jeff Wartes]

There is some automation around this process in the backup commands here:
https://github.com/whitepages/solrcloud_manager


It’s been tested with 5.4, and will restore arbitrary replication factors. 
Ever assuming the shared filesystem for backups, of course.



On 4/5/16, 3:18 AM, "Reth RM" <reth.ik...@gmail.com> wrote:

>Yes. It should be backing up each shard leader of collection. For each
>collection, for each shard, find the leader and request a backup command on
>that. Further, restore this on new collection, in its respective shard and
>then go on adding new replica which will duly pull it from the newly added
>shards.
>
>
>On Mon, Apr 4, 2016 at 10:32 PM, Zisis Tachtsidis <zist...@runbox.com>
>wrote:
>
>> I've tested backup/restore successfully in a SolrCloud installation with a
>> single node (no replicas). This has been achieved in
>> https://issues.apache.org/jira/browse/SOLR-6637
>> Can you do something similar when more replicas are involved? What I'm
>> looking for is a restore command that will restore index in all replicas of
>> a collection.
>> Judging from the code in /ReplicationHandler.java/ and
>> https://issues.apache.org/jira/browse/SOLR-5750 I assume that more work
>> needs to be done to achieve this.
>>
>> Is my understanding correct? If the situation is like this I guess an
>> alternative would be to just create a new collection, restore index and
>> then
>> add replicas. (I'm using Solr 5.5.0)
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/SolrCloud-backup-restore-tp4267954.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>

Re: SolrCloud backup/restore

[Reth RM]

Yes. It should be backing up each shard leader of collection. For each
collection, for each shard, find the leader and request a backup command on
that. Further, restore this on new collection, in its respective shard and
then go on adding new replica which will duly pull it from the newly added
shards.


On Mon, Apr 4, 2016 at 10:32 PM, Zisis Tachtsidis <zist...@runbox.com>
wrote:

> I've tested backup/restore successfully in a SolrCloud installation with a
> single node (no replicas). This has been achieved in
> https://issues.apache.org/jira/browse/SOLR-6637
> Can you do something similar when more replicas are involved? What I'm
> looking for is a restore command that will restore index in all replicas of
> a collection.
> Judging from the code in /ReplicationHandler.java/ and
> https://issues.apache.org/jira/browse/SOLR-5750 I assume that more work
> needs to be done to achieve this.
>
> Is my understanding correct? If the situation is like this I guess an
> alternative would be to just create a new collection, restore index and
> then
> add replicas. (I'm using Solr 5.5.0)
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/SolrCloud-backup-restore-tp4267954.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

SolrCloud backup/restore

[Zisis Tachtsidis]

I've tested backup/restore successfully in a SolrCloud installation with a
single node (no replicas). This has been achieved in
https://issues.apache.org/jira/browse/SOLR-6637 
Can you do something similar when more replicas are involved? What I'm
looking for is a restore command that will restore index in all replicas of
a collection.
Judging from the code in /ReplicationHandler.java/ and
https://issues.apache.org/jira/browse/SOLR-5750 I assume that more work
needs to be done to achieve this.

Is my understanding correct? If the situation is like this I guess an
alternative would be to just create a new collection, restore index and then
add replicas. (I'm using Solr 5.5.0)



--
View this message in context: 
http://lucene.472066.n3.nabble.com/SolrCloud-backup-restore-tp4267954.html
Sent from the Solr - User mailing list archive at Nabble.com.