Re: Live index upgrading

2019-06-21 Thread Erick Erickson
You’re exactly right that storing all the fields necessary to reconstruct the document is a way to not have to reindex from scratch. Of course that bloats your index, in large installations perhaps unacceptably. bq. Is there a convenient place to store…. Lucene itself doesn’t preserve anything

Re: Live index upgrading

2019-06-21 Thread David Allouche
The bottom line for me, is that I am not going to upgrade to Lucene8 for a while. The index migration would either cause a service interruption, or would require a little while to implement. I have more urgent technical debt to deal with. > On 21 Jun 2019, at 19:11, David Allouche wrote: >

Re: Live index upgrading

2019-06-21 Thread David Allouche
Unfortunately, I cannot assume SolrCloud, because our software predates Solr. So I would either need to switch to Solr or reimplement a work-around for the lack of index migration. I am reluctant to switch to Solr because it increases the operational complexity. I understand the argument: if th

Re: Live index upgrading

2019-06-21 Thread Erick Erickson
Assuming SolrCloud, reindex from scratch into a new collection then use collection aliasing when you were ready to switch. You don’t need to stop your clients when you use CREATEALIAS. Prior to writing the marker, Lucene would appear to work with older indexes, but there would be subtle errors

Re: Live index upgrading

2019-06-21 Thread David Allouche
Wow. That is annoying. What is the reason for this? I assumed there was a smooth upgrade path, but apparently, by design, one has to rebuild the index at least once every two major releases. So, my question becomes, what is the recommended way of dealing with reindex-from-scratch without servic

Re: Live index upgrading

2019-06-17 Thread Erick Erickson
Let’s back up a bit. What version of Lucene are you using? Starting with Lucene 8, any index that’s ever been touched by Lucene 6 will not open. It does not matter if the index has been completely rewritten. It does not matter if it’s been run through IndexUpgraderTool, which just does a forceMe

Live index upgrading

2019-06-17 Thread David Allouche
Hello, I use Lucene with PyLucene on a public-facing web application. We have a moderately large index (~24M documents, ~11GB index data), with a constant stream of new documents. I recently upgraded to PyLucene 7. When trying to test the new release of PyLucene 8, I encountered an IndexForma