I have a PR with a test and a possible fix for this: https://github.com/apache/lucene/pull/12326, anyone for review?
Tomás On Tue, Dec 14, 2021 at 7:11 AM Ian Lea <ian....@gmail.com> wrote: > Thanks for the response. > https://issues.apache.org/jira/browse/LUCENE-10314 > > Will we still be able to decide, maybe years down the line, that we do want > to search on fieldX after all, and be able to change the code and reindex > the maybe small proportion of documents that have a value for fieldX > without having to create a new index from scratch? That happens. > > > -- > Ian. > > > On Tue, Dec 14, 2021 at 1:54 PM Adrien Grand <jpou...@gmail.com> wrote: > > > This looks related to the new changes around schema validation. Lucene > > now requires a field to either be absent from a document or be indexed > > with the exact same options (index options, points dimensions, norms, > > doc values type, etc.) as already indexed documents that also have > > this field. > > > > However it's a bug that Lucene fails to open an index that was legal > > in Lucene 8. Can you file a JIRA issue? > > > > On Mon, Dec 13, 2021 at 4:23 PM Ian Lea <ian....@gmail.com> wrote: > > > > > > Hi > > > > > > > > > We have a long-standing index with some mandatory fields and some > > optional > > > fields that has been through multiple lucene upgrades without a full > > > rebuild and on testing out an upgrade from version 8.11.0 to 9.0.0, > when > > > open an IndexWriter we are hitting the exception > > > > > > Exception in thread "main" java.lang.IllegalArgumentException: cannot > > > change field "language" from index options=NONE to inconsistent index > > > options=DOCS > > > at > > > > > > org.apache.lucene.index.FieldInfo.verifySameIndexOptions(FieldInfo.java:245) > > > at > > > > > > org.apache.lucene.index.FieldInfos$FieldNumbers.verifySameSchema(FieldInfos.java:421) > > > at > > > > > > org.apache.lucene.index.FieldInfos$FieldNumbers.addOrGet(FieldInfos.java:357) > > > at > > > > > > org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1263) > > > at > > org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1116) > > > > > > Where language is one of our optional fields. > > > > > > Presumably this is at least somewhat related to "Index options can no > > > longer be changed dynamically" as mentioned at > > > https://lucene.apache.org/core/9_0_0/MIGRATE.html although it fails > > before > > > our code attempts to update the index, and we are not trying to change > > any > > > index options. > > > > > > Adding some displays to IndexWriter and FieldInfos and logging rather > > than > > > throwing the exception I see > > > > > > language curr=NONE, other=NONE > > > language curr=NONE, other=NONE > > > language curr=NONE, other=NONE > > > language curr=NONE, other=NONE > > > language curr=NONE, other=NONE > > > language curr=NONE, other=NONE > > > language curr=NONE, other=NONE > > > language curr=NONE, other=NONE > > > language curr=NONE, other=DOCS > > > language curr=NONE, other=NONE > > > language curr=NONE, other=NONE > > > language curr=NONE, other=NONE > > > language curr=NONE, other=NONE > > > language curr=NONE, other=NONE > > > language curr=NONE, other=NONE > > > language curr=NONE, other=NONE > > > language curr=NONE, other=NONE > > > language curr=NONE, other=NONE > > > language curr=NONE, other=DOCS > > > language curr=NONE, other=DOCS > > > language curr=NONE, other=DOCS > > > language curr=NONE, other=DOCS > > > language curr=NONE, other=DOCS > > > language curr=NONE, other=DOCS > > > language curr=NONE, other=DOCS > > > language curr=NONE, other=DOCS > > > > > > where there is one line per segment. It logs the exception whenever > > > other=DOCS. Subset with segment info: > > > > > > segment _x8(8.2.0):c31753/-1:[diagnostics={timestamp=1565623850605, > > > lucene.version=8.2.0, java.vm.version=11.0.3+7, java.version=11.0.3, > > > mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop, > > > java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10, > > > java.runtime.version=11.0.3+7, > > > os=Linux}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}] > > > > > > language curr=NONE, other=NONE > > > > > > segment _y9(8.7.0):c43531/-1:[diagnostics={timestamp=1604597581562, > > > lucene.version=8.7.0, java.vm.version=11.0.3+7, java.version=11.0.3, > > > mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop, > > > java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10, > > > java.runtime.version=11.0.3+7, > > > os=Linux}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}] > > > > > > language curr=NONE, other=DOCS > > > > > > NOT throwing java.lang.IllegalArgumentException: cannot change field > > > "language" from index options=NONE to inconsistent index options=DOCS > > > > > > > > > Some variation on an old-fashioned not set versus not present bug > > perhaps? > > > > > > > > > -- > > > Ian. > > > > > > > > -- > > Adrien > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > >
