Hi Rahul, This exception complains that IndexingChain did not deduplicate terms as expected.
I don't recall seeing this exception before (which doesn't mean it's not a real bug). What JVM are you running? Does this exception frequently occur or was it a one-off? On Thu, Sep 28, 2023 at 4:49 PM Rahul Goswami <rahul196...@gmail.com> wrote: > > Hi, > Following up on my issue...anyone who's seen similar exceptions ? Or any > insights on what might be going on? > > Thanks, > Rahul > > On Wed, Sep 27, 2023 at 1:00 AM Rahul Goswami <rahul196...@gmail.com> wrote: > > > Hello, > > On one of the servers running Solr 7.7.2, during indexing I observe 2 > > different kinds of exceptions coming from the Lucene codec layer. I can't > > think of an application/data issue that could be causing this. > > > > In particular, Exception-2 seems like a potential bug since it complains > > about "terms out of order" even though both byte arrays are essentially the > > same. Reason I say this is that the FutureArrays.mismatch() is supposed to > > behave like Java's Arrays.mismatch which returns -1 if NO mismatch is > > found. However the check in the below line treats the value -1 as a > > mismatch causing the exception. > > > > > > https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.7.2/lucene/core/src/java/org/apache/lucene/util/StringHelper.java#L46 > > > > Happy to submit a PR for this if there is a consensus on this being a bug. > > > > Would appreciate any inputs on the exceptions seen below! > > > > *Exception-1:* > > > > 2023-09-19 10:13:48.901 ERROR (qtp1859039536-1691) [ > > x:fsindex_FileIndexer20234799_shard_1] o.a.s.s.HttpSolrCall > > null:org.apache.solr.common.SolrException: Server error writing document id > > 6182!bbdbe92468734899c738f048e6f58245 to the index > > at > > org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:240) > > at > > org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67) > > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:1002) > > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:1233) > > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.lambda$versionAdd$2(DistributedUpdateProcessor.java:1082) > > at org.apache.solr.update.VersionBucket.runWithLock(VersionBucket.java:50) > > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1082) > > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:694) > > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > at > > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118) > > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > at > > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118) > > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > at > > org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:92) > > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > at > > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118) > > at > > org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:261) > > at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:188) > > at > > org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97) > > at > > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68) > > at > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:202) > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551) > > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711) > > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516) > > at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395) > > at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341) > > at > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) > > at > > com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121) > > at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133) > > at > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) > > at > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) > > at > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) > > at > > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > > at > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > > at > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) > > at > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588) > > at > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) > > at > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) > > at > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) > > at > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) > > at > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557) > > at > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) > > at > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) > > at > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) > > at > > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) > > at > > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) > > at > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > > at > > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > > at > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > > at org.eclipse.jetty.server.Server.handle(Server.java:502) > > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) > > at > > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) > > at > > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) > > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) > > at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) > > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) > > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) > > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) > > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) > > at > > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) > > at > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) > > at > > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) > > at java.base/java.lang.Thread.run(Unknown Source) > > Caused by: org.apache.lucene.store.AlreadyClosedException: this > > IndexWriter is closed > > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:680) > > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:694) > > at > > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1613) > > at > > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1608) > > at > > org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:969) > > at > > org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:341) > > at > > org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:288) > > at > > org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:235) > > ... 83 more > > > > *Caused by: org.apache.lucene.index.CorruptIndexException: docs out of > > order (23 <= 23 ) > > (resource=FSIndexOutput(path="D:\CI-IndexDirectory-2023\fsindex_FileIndexer20234799_shard_1\index\_1h_Lucene50_0.doc"))* > > at > > org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:225) > > at > > org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:141) > > at > > org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:913) > > at > > org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:354) > > at > > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140) > > at > > org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95) > > at > > org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166) > > at > > org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469) > > at > > org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554) > > at > > org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:413) > > at > > org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:514) > > at > > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616) > > ... 88 more > > > > > > *Exception-2* > > > > 2023-09-19 10:13:43.261 ERROR (qtp1859039536-1044) [ > > x:fsindex_FileIndexer20234799_shard_4] o.a.s.s.HttpSolrCall > > null:org.apache.solr.common.SolrException: Server error writing document id > > 1161!07acb2f00d1da463801bc102ecfaee06 to the index > > at > > org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:240) > > at > > org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67) > > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:1002) > > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:1233) > > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.lambda$versionAdd$2(DistributedUpdateProcessor.java:1082) > > at org.apache.solr.update.VersionBucket.runWithLock(VersionBucket.java:50) > > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1082) > > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:694) > > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > at > > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118) > > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > at > > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118) > > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > at > > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118) > > at > > org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:261) > > at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:188) > > at > > org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97) > > at > > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68) > > at > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:202) > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551) > > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711) > > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516) > > at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395) > > at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341) > > at > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) > > at > > com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121) > > at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133) > > at > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) > > at > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) > > at > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) > > at > > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > > at > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > > at > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) > > at > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588) > > at > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) > > at > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) > > at > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) > > at > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) > > at > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557) > > at > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) > > at > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) > > at > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) > > at > > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) > > at > > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) > > at > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > > at > > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) > > at > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > > at org.eclipse.jetty.server.Server.handle(Server.java:502) > > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) > > at > > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) > > at > > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) > > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) > > at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) > > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) > > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) > > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) > > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) > > at > > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) > > at > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) > > at > > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) > > at java.base/java.lang.Thread.run(Unknown Source) > > Caused by: org.apache.lucene.store.AlreadyClosedException: this > > IndexWriter is closed > > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:680) > > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:694) > > at > > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1613) > > at > > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1608) > > at > > org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:969) > > at > > org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:341) > > at > > org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:288) > > at > > org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:235) > > ... 83 more > > > > *Caused by: java.lang.IllegalArgumentException: terms out of order: > > priorTerm=[55 4e 43 2d 4e 54 5f 33 30 2d 66 69 6c 65 2d 30 30 34 5c 43 61 > > 64 75 73 65 72 73 24 5c 4a 42 61 75 65 72 5c 34 20 57 48 4c 20 43 4f 4e 53 > > 4f 4c 49 44 41 54 49 4f 4e 5c 34 77 68 6c 20 6d 65 65 74 69 6e 67 20 6a 70 > > 65 67 73 5c 31 30 2d 33 30 2d 30 38 20 6d 74 67],currentTerm=[55 4e 43 2d > > 4e 54 5f 33 30 2d 66 69 6c 65 2d 30 30 34 5c 43 61 64 75 73 65 72 73 24 5c > > 4a 42 61 75 65 72 5c 34 20 57 48 4c 20 43 4f 4e 53 4f 4c 49 44 41 54 49 4f > > 4e 5c 34 77 68 6c 20 6d 65 65 74 69 6e 67 20 6a 70 65 67 73 5c 31 30 2d 33 > > 30 2d 30 38 20 6d 74 67]* at > > org.apache.lucene.util.StringHelper.bytesDifference(StringHelper.java:47) > > at > > org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.addTermsDict(Lucene70DocValuesConsumer.java:461) > > at > > org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.doAddSortedField(Lucene70DocValuesConsumer.java:437) > > at > > org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.addSortedField(Lucene70DocValuesConsumer.java:394) > > at > > org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addSortedField(PerFieldDocValuesFormat.java:119) > > at > > org.apache.lucene.index.SortedDocValuesWriter.flush(SortedDocValuesWriter.java:163) > > at > > org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:251) > > at > > org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:135) > > at > > org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469) > > at > > org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554) > > at > > org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:417) > > at > > org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:514) > > at > > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616) > > ... 88 more > > > > Thanks, > > Rahul > > -- Adrien --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org