[
https://issues.apache.org/jira/browse/SOLR-14254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gerlowski updated SOLR-14254:
-----------------------------------
Description:
I believe I found a backcompat break between 8.4.1 and 8.3.1.
I encountered this when a Solr 8.3.1 cluster was upgraded to 8.4.1. On 8.4.
nodes, several collections had cores fail to come up with
{{CorruptIndexException}}:
{code}
2020-02-10 20:58:26.136 ERROR
(coreContainerWorkExecutor-2-thread-1-processing-n:192.168.1.194:8983_solr) [
] o.a.s.c.CoreContainer Error waiting for SolrCore to be loaded on startup =>
org.apache.sol
r.common.SolrException: Unable to create core [testbackcompat_shard1_replica_n1]
at
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1313)
org.apache.solr.common.SolrException: Unable to create core
[testbackcompat_shard1_replica_n1]
at
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1313)
~[?:?]
at
org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:788) ~[?:?]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:202)
~[metrics-core-4.0.5.jar:4.0.5]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1072) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
at
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292)
~[?:?]
... 7 more
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2182)
~[?:?]
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2302) ~[?:?]
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1132) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1013) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
at
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292)
~[?:?]
... 7 more
Caused by: org.apache.lucene.index.CorruptIndexException: codec mismatch:
actual codec=Lucene50PostingsWriterDoc vs expected
codec=Lucene84PostingsWriterDoc
(resource=MMapIndexInput(path="/Users/jasongerlowski/run/solrdata/data/testbackcompat_shard1_replica_n1/data/index/_0_FST50_0.doc"))
at
org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:208) ~[?:?]
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:198)
~[?:?]
at
org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255) ~[?:?]
at
org.apache.lucene.codecs.lucene84.Lucene84PostingsReader.<init>(Lucene84PostingsReader.java:82)
~[?:?]
at
org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:66)
~[?:?]
at
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:315)
~[?:?]
at
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:395)
~[?:?]
at
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:114)
~[?:?]
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:84)
~[?:?]
at
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:177)
~[?:?]
at
org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:219)
~[?:?]
at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109)
~[?:?]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526)
~[?:?]
at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:116) ~[?:?]
at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:92) ~[?:?]
at
org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:39)
~[?:?]
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2146)
~[?:?]
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2302) ~[?:?]
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1132) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1013) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
at
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292)
~[?:?]
... 7 more
{code}
Only some cores were affected. After some digging, we noticed that these cores
were unique in that they had a field set up to use Solr's Text Tagger. Not the
use of a non-default postings-format (FST50).
{code}
<fieldType class="solr.TextField" name="tagger" omitNorms="true"
omitTermFreqAndPositions="true" postingsFormat="FST50">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ConcatenateGraphFilterFactory"
preservePositionIncrements="false"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<field name="tagger_field" type="tagger" indexed="true" stored="true" />
{code}
I've since been able to reproduce the problem locally with the following steps:
# Start Solr 8.3.1 with a data directory /some/data/dir
# Modify the default configset (or create a new configset) by adding the
fieldType/field above to your schema.
# Index a document that contains a "tagger_field" value and commit.
# Stop Solr 8.3.1
# Start Solr 8.4.1 pointed at the /some/data/dir as its data directory.
At this point, the core should be down and you should see the
CorruptIndexException in your logs.
This _seems_ related to LUCENE-9027, though I don't know enough about how
Lucene handles ensuring support for older indices to know exactly what happened
there.
was:
I believe I found a backcompat break between 8.4.1 and 8.3.1.
I encountered this when a Solr 8.3.1 cluster was upgraded to 8.4.1. On 8.4.
nodes, several collections had cores fail to come up with
{{CorruptIndexException}}:
{code}
2020-02-10 20:58:26.136 ERROR
(coreContainerWorkExecutor-2-thread-1-processing-n:192.168.1.194:8983_solr) [
] o.a.s.c.CoreContainer Error waiting for SolrCore to be loaded on startup =>
org.apache.sol
r.common.SolrException: Unable to create core [testbackcompat_shard1_replica_n1]
at
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1313)
org.apache.solr.common.SolrException: Unable to create core
[testbackcompat_shard1_replica_n1]
at
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1313)
~[?:?]
at
org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:788) ~[?:?]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:202)
~[metrics-core-4.0.5.jar:4.0.5]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1072) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
at
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292)
~[?:?]
... 7 more
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2182)
~[?:?]
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2302) ~[?:?]
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1132) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1013) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
at
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292)
~[?:?]
... 7 more
Caused by: org.apache.lucene.index.CorruptIndexException: codec mismatch:
actual codec=Lucene50PostingsWriterDoc vs expected
codec=Lucene84PostingsWriterDoc
(resource=MMapIndexInput(path="/Users/jasongerlowski/run/solrdata/data/testbackcompat_shard1_replica_n1/data/index/_0_FST50_0.doc"))
at
org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:208) ~[?:?]
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:198)
~[?:?]
at
org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255) ~[?:?]
at
org.apache.lucene.codecs.lucene84.Lucene84PostingsReader.<init>(Lucene84PostingsReader.java:82)
~[?:?]
at
org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:66)
~[?:?]
at
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:315)
~[?:?]
at
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:395)
~[?:?]
at
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:114)
~[?:?]
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:84)
~[?:?]
at
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:177)
~[?:?]
at
org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:219)
~[?:?]
at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109)
~[?:?]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526)
~[?:?]
at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:116) ~[?:?]
at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:92) ~[?:?]
at
org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:39)
~[?:?]
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2146)
~[?:?]
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2302) ~[?:?]
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1132) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1013) ~[?:?]
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
at
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292)
~[?:?]
... 7 more
{code}
Only some cores were affected. After some digging, we noticed that these cores
were unique in that they had a field set up to use Solr's Text Tagger:
{code}
<fieldType class="solr.TextField" name="tagger" omitNorms="true"
omitTermFreqAndPositions="true" postingsFormat="FST50">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ConcatenateGraphFilterFactory"
preservePositionIncrements="false"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<field name="tagger_field" type="tagger" indexed="true" stored="true" />
{code}
I've since been able to reproduce the problem locally with the following steps:
# Start Solr 8.3.1 with a data directory /some/data/dir
# Modify the default configset (or create a new configset) by adding the
fieldType/field above to your schema.
# Index a document that contains a "tagger_field" value and commit.
# Stop Solr 8.3.1
# Start Solr 8.4.1 pointed at the /some/data/dir as its data directory.
At this point, the core should be down and you should see the
CorruptIndexException in your logs.
This _seems_ related to LUCENE-9027, though I don't know enough about how
Lucene handles ensuring support for older indices to know exactly what happened
there.
> Index backcompat break between 8.3.1 and 8.4.1
> ----------------------------------------------
>
> Key: SOLR-14254
> URL: https://issues.apache.org/jira/browse/SOLR-14254
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Jason Gerlowski
> Priority: Major
>
> I believe I found a backcompat break between 8.4.1 and 8.3.1.
> I encountered this when a Solr 8.3.1 cluster was upgraded to 8.4.1. On 8.4.
> nodes, several collections had cores fail to come up with
> {{CorruptIndexException}}:
> {code}
> 2020-02-10 20:58:26.136 ERROR
> (coreContainerWorkExecutor-2-thread-1-processing-n:192.168.1.194:8983_solr) [
> ] o.a.s.c.CoreContainer Error waiting for SolrCore to be loaded on startup
> => org.apache.sol
> r.common.SolrException: Unable to create core
> [testbackcompat_shard1_replica_n1]
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1313)
> org.apache.solr.common.SolrException: Unable to create core
> [testbackcompat_shard1_replica_n1]
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1313)
> ~[?:?]
> at
> org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:788)
> ~[?:?]
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:202)
> ~[metrics-core-4.0.5.jar:4.0.5]
> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> ~[?:?]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1072) ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292)
> ~[?:?]
> ... 7 more
> Caused by: org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2182)
> ~[?:?]
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2302)
> ~[?:?]
> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1132)
> ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1013) ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292)
> ~[?:?]
> ... 7 more
> Caused by: org.apache.lucene.index.CorruptIndexException: codec mismatch:
> actual codec=Lucene50PostingsWriterDoc vs expected
> codec=Lucene84PostingsWriterDoc
> (resource=MMapIndexInput(path="/Users/jasongerlowski/run/solrdata/data/testbackcompat_shard1_replica_n1/data/index/_0_FST50_0.doc"))
> at
> org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:208)
> ~[?:?]
> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:198)
> ~[?:?]
> at
> org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255) ~[?:?]
> at
> org.apache.lucene.codecs.lucene84.Lucene84PostingsReader.<init>(Lucene84PostingsReader.java:82)
> ~[?:?]
> at
> org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:66)
> ~[?:?]
> at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:315)
> ~[?:?]
> at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:395)
> ~[?:?]
> at
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:114)
> ~[?:?]
> at
> org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:84) ~[?:?]
> at
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:177)
> ~[?:?]
> at
> org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:219)
> ~[?:?]
> at
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109)
> ~[?:?]
> at
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526) ~[?:?]
> at
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:116) ~[?:?]
> at
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:92) ~[?:?]
> at
> org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:39)
> ~[?:?]
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2146)
> ~[?:?]
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2302)
> ~[?:?]
> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1132)
> ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1013) ~[?:?]
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
> at
> org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292)
> ~[?:?]
> ... 7 more
> {code}
> Only some cores were affected. After some digging, we noticed that these
> cores were unique in that they had a field set up to use Solr's Text Tagger.
> Not the use of a non-default postings-format (FST50).
> {code}
> <fieldType class="solr.TextField" name="tagger" omitNorms="true"
> omitTermFreqAndPositions="true" postingsFormat="FST50">
> <analyzer type="index">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.EnglishPossessiveFilterFactory"/>
> <filter class="solr.ASCIIFoldingFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.ConcatenateGraphFilterFactory"
> preservePositionIncrements="false"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.EnglishPossessiveFilterFactory"/>
> <filter class="solr.ASCIIFoldingFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
> <field name="tagger_field" type="tagger" indexed="true" stored="true"
> />
> {code}
> I've since been able to reproduce the problem locally with the following
> steps:
> # Start Solr 8.3.1 with a data directory /some/data/dir
> # Modify the default configset (or create a new configset) by adding the
> fieldType/field above to your schema.
> # Index a document that contains a "tagger_field" value and commit.
> # Stop Solr 8.3.1
> # Start Solr 8.4.1 pointed at the /some/data/dir as its data directory.
> At this point, the core should be down and you should see the
> CorruptIndexException in your logs.
> This _seems_ related to LUCENE-9027, though I don't know enough about how
> Lucene handles ensuring support for older indices to know exactly what
> happened there.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
