ctargett commented on a change in pull request #594: SOLR-13259: Add new section on Reindexing in Solr URL: https://github.com/apache/lucene-solr/pull/594#discussion_r262133903
########## File path: solr/solr-ref-guide/src/reindexing.adoc ########## @@ -0,0 +1,185 @@ += Reindexing +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +There are several types of changes to Solr configuration that require you to reindex your data. + +These changes include editing properties of fields or field types; adding fields, field types, or copy field rules; +upgrading Solr; and some system configuration properties. + +It's important to be aware that many changes require reindexing, because there are times when not reindexing +can have negative consequences for Solr as a system, or for the ability of your users to find what they are looking for. + +There is no process in Solr for programmatically reindexing data. When we say "reindex", we mean, literally, +"index it again". However you got the data into the index the first time, you will run that process again. +It is strongly recommended that Solr users index their data in a repeatable, consistent way, so that the process can be +easily repeated when the need for reindexing arises. + +Reindexing is recommended during major upgrades, so in addition to covering what types of configuration changes should trigger a reindex, this section will also cover strategies for reindexing. + +== Changes that Require Reindex + +=== Schema Changes + +All changes to a collection's schema require reindexing. This is because many of the available options are only +applied during the indexing process. Solr simply has no way to implement the desired change without reindexing +the data. + +To understand the general reason why reindexing is ever required, it's helpful to understand the relationship between +Solr's schema and the underlying Lucene index. Lucene does not use a schema, it is a Solr-only concept. When you delete +a field from Solr's schema, it does not modify Lucene's index in any way. When you add a field to Solr's schema, the +field does not exist in Lucene's index until a document that contains the field is indexed. + +This means that there are many types of schema changes that cannot be reflected in the index simply by modifying +Solr's schema. This is different from most database models where schemas are used. With regard to indexing, Solr's +schema acts like a rulebook for indexing documents by telling Lucene how to interpret the data being sent. Once the +documents are in Lucene, Solr's schema has no control over the underlying data structure. + +In addition to the types of schema changes described in the following sections, changing the schema `version` property +is equivalent to changing field type properties. This type of change is usually only made during or because of a major upgrade. + +==== Adding or Deleting Fields + +If you add or delete a field from Solr's schema, it's strongly recommended to reindex. + +When you add a field, you generally do so with the intent to use the field in some way. +Since documents were indexed before the field was added, the index will not hold any references to the field for earlier documents. +If you want to use the new field for faceting, for example, the new field facet will not include any documents that were not indexed with the new field. + +There is a slightly different situation when deleting a field. +In this case, since simply removing the field from the schema doesn't change anything about the index, the field will still be in the index until the documents are reindexed. +In fact, Lucene may keep a reference to a deleted field _forever_ (see also https://issues.apache.org/jira/browse/LUCENE-1761[LUCENE-1761]). +This may only be an issue for your environment if you try to add a field that has the same name as a deleted field, +but it can also be an issue for dynamic field rules that are later removed. + +==== Changing Field and Field Type Field Properties + +Solr has two ways of defining field properties. + +The first is to define properties on a field type. These properties are then applied to all fields of that type unless they are explicitly overriden. + +The second is an override to a property inherited from the field type defined on the field itself. + +If a property has been defined for a field type but the property is not overridden by defining a different value for the +property for a field, then changing the property on the field type is equivalent to changing it on the field itself. + +Changes to *any* field/field type property described in <<field-type-definitions-and-properties.adoc#field-type-properties,Field Type Properties>> require reindexing in order for the change to be reflected in all documents. +The list of changes that require reindexing includes: + +* Changing a field from stored to not stored, and vice versa. +* Changing a field from indexed to not indexed, and vice versa. +* Changing a field from multi-valued to single-valued, and vice versa. +* <<Changing Field Analysis>> + +Be sure to reference the Field Type Properties section linked above for the complete list of properties that would require a reindex. + +In some cases, it can be possible to change a field/field type property value and it will only apply to documents +indexed _after_ the change. This is not recommended to ensure consistent behavior, but may be acceptable for your +use case as a temporary condition before a full reindexing can be scheduled. + +==== Changing Field Analysis + +Beyond specific field-level properties, <<analyzers.adoc#analyzers,analysis chains>> are also configured on field types, and are applied at index and/or query time. + +It's possible to define separate analysis chains for indexing and query events, or you can define a single chain +that is applied to both event types. + +If you change the analysis chain that applies to indexing events, it is strongly recommended that you reindex. +This is because all of the changes that occur due to the chain configuration are applied to documents as they are +being indexed, and only reindexing will allow your changes to take effect on documents. + +While not reindexing after analyzer changes is not required, be aware that not reindexing can cause unexpected +query results in many cases. + +For example, if you indexed a number of documents and then decide you'd like to use the `LowerCaseTokenizerFactory` +to ensure all text is converted to lower case, you will have a mix of entries in the field: some in their original +case ("iPhone"), and newer documents in all lower-case ("iphone"). If you do not reindex the original set of documents, +a query such as "iphone" will not match documents with "iPhone", because the schema rules enforce lower case on the +query, but that's not what is in the index. + +The only time you do not have to reindex when changing a field type's analysis chain is when the changes impact +queries *only* (and you know that you do not need to make corresponding changes to the index analysis). + +=== Solrconfig Changes + +Only one parameter change to Solr's `solrconfig.xml` requires reindexing. That parameter is the `luceneMatchVersion`, +which controls the compatibility of Solr with Lucene changes. Since this parameter can change the rules for analysis behind the scenes, it's always recommended to reindex when changing this value. Usually, however, this is only changed in conjunction with a major upgrade. + +However, if you make a change to Solr's <<update-request-processors.adoc#update-request-processors,Update Request Processors>>, it's generally because you want to change something about how _update requests_ (documents) are _processed_ (indexed). In this case, you can decide based on the change if you want to reindex your documents to implement the changes you've made. + +Similarly, if you change the `codecFactory` parameter in `solrconfig.xml`, it is again strongly recommended that you +plan to reindex your documents to avoid unintended behavior. + +== Upgrades + +When upgrading between major versions (for example, from a 7.x release to 8.0 or 8.x), a best practice +is to always reindex your data. +The reason for this is that subtle changes may occur in default field type definitions or the underlying code. + +[NOTE] +If you have *not* changed your schema as part of an upgrade from one minor release to another (such as, from 7.x +to 7.x), you can often get away with reindexing your documents without first deleting your index. However, when Review comment: Was supposed to be 'not upgrading'. Also made the versions more clear ('a later 7.x release') ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
