[
https://issues.apache.org/jira/browse/SOLR-14254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034015#comment-17034015
]
David Smiley edited comment on SOLR-14254 at 2/10/20 11:35 PM:
---------------------------------------------------------------
CC [~jpountz] I think Hossman has a point that the "FST50" name should have
changed when the index version changed to like FST84 if 8.4 has a different
version than prior. It'd be cool if a user could say postingsFormat="FST";
clearly out of scope I know.
Aaaaanyway... Cassandra, to answer your question: upgrade advise is re-index if
that's easiest. Another option is for a user to stay at the previous version
and change the postingsFormat to be non-existent to be the default and then
trigger an optimize which will rewrite it to something that will survive to the
next version. Then upgrade the software. Then to get back to an FST based
format, change the postingsFormat config back then do an optimize again.
Wether or not users should use postingsFormat=FST50 or let it default is a
trade-off in performance vs compatibility. FST kicks but but perhaps you won't
notice if you send small amounts of text (e.g. queries). If someone is
desperate, the old code can be moved into Solr and packaged up as a plugin.
was (Author: dsmiley):
CC [~jpountz] I think Hossman has a point that the "FST50" name should have
changed when the index version changed to like FST84 if 8.4 has a different
version than prior. It'd be cool if a user could say postingsFormat="FST";
clearly out of scope I know.
Aaaaanyway... Cassandra, to answer your question: upgrade advise is re-index if
that's easiest. Another option is for a user to stay at the previous version
and change the postingsFormat to be non-existent to be the default and then
trigger an optimize which will rewrite it to something that will survive to the
next version. Then upgrade the software, then change the version back
(optionally). Wether or not users should use postingsFormat=FST50 or let it
default is a trade-off in performance vs compatibility. FST kicks but but
perhaps you won't notice if you send small amounts of text (e.g. queries). If
someone is desperate, the old code can be moved into Solr and packaged up as a
plugin.
> Index backcompat break between 8.3.1 and 8.4.1
> ----------------------------------------------
>
> Key: SOLR-14254
> URL: https://issues.apache.org/jira/browse/SOLR-14254
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Jason Gerlowski
> Priority: Major
>
> I believe I found a backcompat break between 8.4.1 and 8.3.1.
> I encountered this when a Solr 8.3.1 cluster was upgraded to 8.4.1. On 8.4.
> nodes, several collections had cores fail to come up with
> {{CorruptIndexException}}:
> {code}
> 2020-02-10 20:58:26.136 ERROR
> (coreContainerWorkExecutor-2-thread-1-processing-n:192.168.1.194:8983_solr) [
> ] o.a.s.c.CoreContainer Error waiting for SolrCore to be loaded on startup
> => org.apache.sol
> r.common.SolrException: Unable to create core
> [testbackcompat_shard1_replica_n1]
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1313)
> org.apache.solr.common.SolrException: Unable to create core
> [testbackcompat_shard1_replica_n1]
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1313)
> ~[?:?]
> at
> org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:788)
> ~[?:?]
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:202)
> ~[metrics-core-4.0.5.jar:4.0.5]
> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> ~[?:?]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1072) ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292)
> ~[?:?]
> ... 7 more
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2182)
> ~[?:?]
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2302)
> ~[?:?]
> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1132)
> ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1013) ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292)
> ~[?:?]
> ... 7 more
> Caused by: org.apache.lucene.index.CorruptIndexException: codec mismatch:
> actual codec=Lucene50PostingsWriterDoc vs expected
> codec=Lucene84PostingsWriterDoc
> (resource=MMapIndexInput(path="/Users/jasongerlowski/run/solrdata/data/testbackcompat_shard1_replica_n1/data/index/_0_FST50_0.doc"))
> at
> org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:208)
> ~[?:?]
> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:198)
> ~[?:?]
> at
> org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255) ~[?:?]
> at
> org.apache.lucene.codecs.lucene84.Lucene84PostingsReader.<init>(Lucene84PostingsReader.java:82)
> ~[?:?]
> at
> org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:66)
> ~[?:?]
> at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:315)
> ~[?:?]
> at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:395)
> ~[?:?]
> at
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:114)
> ~[?:?]
> at
> org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:84) ~[?:?]
> at
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:177)
> ~[?:?]
> at
> org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:219)
> ~[?:?]
> at
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109)
> ~[?:?]
> at
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526) ~[?:?]
> at
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:116) ~[?:?]
> at
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:92) ~[?:?]
> at
> org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:39)
> ~[?:?]
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2146)
> ~[?:?]
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2302)
> ~[?:?]
> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1132)
> ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1013) ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292)
> ~[?:?]
> ... 7 more
> {code}
> Only some cores were affected. After some digging, we noticed that these
> cores were unique in that they had a field set up to use Solr's Text Tagger.
> Not the use of a non-default postings-format (FST50).
> {code}
> <fieldType class="solr.TextField" name="tagger" omitNorms="true"
> omitTermFreqAndPositions="true" postingsFormat="FST50">
> <analyzer type="index">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.EnglishPossessiveFilterFactory"/>
> <filter class="solr.ASCIIFoldingFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.ConcatenateGraphFilterFactory"
> preservePositionIncrements="false"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.EnglishPossessiveFilterFactory"/>
> <filter class="solr.ASCIIFoldingFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <field name="tagger_field" type="tagger" indexed="true" stored="true"
> />
> {code}
> I've since been able to reproduce the problem locally with the following
> steps:
> # Start Solr 8.3.1 with a data directory /some/data/dir
> # Modify the default configset (or create a new configset) by adding the
> fieldType/field above to your schema.
> # Index a document that contains a "tagger_field" value and commit.
> # Stop Solr 8.3.1
> # Start Solr 8.4.1 pointed at the /some/data/dir as its data directory.
> At this point, the core should be down and you should see the
> CorruptIndexException in your logs.
> This _seems_ related to LUCENE-9027, though I don't know enough about how
> Lucene handles ensuring support for older indices to know exactly what
> happened there.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
