Re: Is it safe to upgrade an existing field to docvalues?

Hi Ronald, Turning on docValues for existing field works in Solr 4. As you mentioned it will use un-inverting method if docValues are nit found on existing document. This all works fine until segments that have documents without docValues merge with segment that have docValues for the field. In

Re: Is it safe to upgrade an existing field to docvalues?

Of course I see your point Ronald, and don't get me wrong, I don't think it is a bad idea. I simply think can bring some complexity and confusion if we start to use it as a common approach. Anyway let's see what the other Solr gurus think :) Cheers On Thu, Aug 25, 2016 at 2:21 PM, Ronald Wood

Re: Is it safe to upgrade an existing field to docvalues?

Thanks, Toke. I’m still surveying the code; do you know of a place in the code that might be more problematic? We’d be mainly concerned about searching, sorting and (simple, low-cardinality) faceting working for us. Some features like grouping are not currently used by us, so in a pinch a

Re: Is it safe to upgrade an existing field to docvalues?

Alessandro, yes I can see how this could be conceived of as a more general problem; and yes useDocValues also strikes me as being unlike the other properties since it would only be used temporarily. We’ve actually had to migrate fields from one to another when changing types, along with

Re: Is it safe to upgrade an existing field to docvalues?

Ronald Wood wrote: > Did you find you had to do a full conversion all at once because simply > turning on > docvalues in the schema caused issues? Yes. > I ask because my presupposition has been that we could turn it on without any > harm as we incrementally converted our

Re: Is it safe to upgrade an existing field to docvalues?

> switching is done in Solr on field.hasDocValues. The code would be amended > to (field.hasDocValues && field.useDocValues) throughout. > This is correct. Currently we use DocValues if they are available, and to check the availabilty we check the schema attribute. This can be problematic in the

Re: Is it safe to upgrade an existing field to docvalues?

Alessandro Benedetti wrote: > So basically using your tool you build a copy of the index ( similar to > what optimize does) without affecting the main index right ? Yes. Your step-by-step is spot on. In the end we re-indexed everything, because there were other issues

Re: Is it safe to upgrade an existing field to docvalues?

I created https://issues.apache.org/jira/browse/SOLR-9437 for the proposal below. I suppose beside feasibility, there’s the question of whether the change is needed by others. I’d love to hear if it meets anyone else’s needs. - Ronald S. Wood On 8/24/16, 15:08, "Ronald Wood"

Re: Is it safe to upgrade an existing field to docvalues?

OK. Thank you, Alessandro, for clarifying this matter. The reason I wasn’t sure about this is that this is somewhat ambiguous in the documentation. In the 6.1 Guide I see: “If you have already indexed data into your Solr index, you will need to completely re-index your content after changing

Re: Is it safe to upgrade an existing field to docvalues?

I am sorry Ronald but : " ask because my presupposition has been that we could turn it on without any harm as we incrementally converted our indexes." This is not possible, if you change the schema and then slowly update the documents you are introducing inconsistency that will reflect in

Re: Is it safe to upgrade an existing field to docvalues?

Yes, Shawn, our indexes meet the requirements for atomic updates. We actually depend on atomic updates since our users can alter metadata about any of our indexed records. We don’t have to incur the cost of a full re-index of a record for every change. This is especially critical when a user

Re: Is it safe to upgrade an existing field to docvalues?

On 8/23/2016 2:01 PM, Ronald Wood wrote: > In general, is there a way to migrate existing indexes (we have petabytes of > data) by enabling docvalues and incrementally re-indexing? We expect the > latter would take a month using an atomic update process. One way to handle it is to build a new

Re: Is it safe to upgrade an existing field to docvalues?

Thanks Toke. I’ve read some of your other helpful blog entries but I missed that one. Did you find you had to do a full conversion all at once because simply turning on docvalues in the schema caused issues? I ask because my presupposition has been that we could turn it on without any harm

Re: Is it safe to upgrade an existing field to docvalues?

Hi Toke ! Good stuff ! So basically using your tool you build a copy of the index ( similar to what optimize does) without affecting the main index right ? So the procedure would be : 1) Solr is running 2) Run the tool pointing to the Solr Index, this will be slow but will generate a new index

Re: Is it safe to upgrade an existing field to docvalues?

On Tue, 2016-08-23 at 20:01 +, Ronald Wood wrote: > In general, is there a way to migrate existing indexes (we have > petabytes of data) by enabling docvalues and incrementally re- > indexing? We expect the latter would take a month using an atomic > update process. I did write a tool for

Is it safe to upgrade an existing field to docvalues?

We are planning to migrate from Solr 4.10.4 to 5.5.2 in the next couple of months. We do not use SolrCloud. When doing initial testing in our dev and qa environments we ran into cases where we got errors for fields that had docvalues newly enabled, but not re-indexed. Mixed