Re: Exception from the codec layer during indexing
Thanks Adrien. I am using Java 17. Occurs like clockwork in this particular customer's environment even after deleting the index and reconstructing. I have never seen this issue myself despite running the same Solr and Java version on more than 100 other nodes. I have no clue about the first exception, but looking at the second one does look like a code bug to me for the reasons mentioned in my email. Also, the storage this customer is running has Software RAID5 . Not sure if that could have anything to do with how the bytes are read from the disk. Or does this exception point to issues with term deduplication in memory before making it to disk? Thanks, Rahul On Thu, Sep 28, 2023 at 3:49 PM Adrien Grand wrote: > Hi Rahul, > > This exception complains that IndexingChain did not deduplicate terms > as expected. > > I don't recall seeing this exception before (which doesn't mean it's > not a real bug). > > What JVM are you running? Does this exception frequently occur or was > it a one-off? > > On Thu, Sep 28, 2023 at 4:49 PM Rahul Goswami > wrote: > > > > Hi, > > Following up on my issue...anyone who's seen similar exceptions ? Or any > > insights on what might be going on? > > > > Thanks, > > Rahul > > > > On Wed, Sep 27, 2023 at 1:00 AM Rahul Goswami > wrote: > > > > > Hello, > > > On one of the servers running Solr 7.7.2, during indexing I observe 2 > > > different kinds of exceptions coming from the Lucene codec layer. I > can't > > > think of an application/data issue that could be causing this. > > > > > > In particular, Exception-2 seems like a potential bug since it > complains > > > about "terms out of order" even though both byte arrays are > essentially the > > > same. Reason I say this is that the FutureArrays.mismatch() is > supposed to > > > behave like Java's Arrays.mismatch which returns -1 if NO mismatch is > > > found. However the check in the below line treats the value -1 as a > > > mismatch causing the exception. > > > > > > > > > > https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.7.2/lucene/core/src/java/org/apache/lucene/util/StringHelper.java#L46 > > > > > > Happy to submit a PR for this if there is a consensus on this being a > bug. > > > > > > Would appreciate any inputs on the exceptions seen below! > > > > > > *Exception-1:* > > > > > > 2023-09-19 10:13:48.901 ERROR (qtp1859039536-1691) [ > > > x:fsindex_FileIndexer20234799_shard_1] o.a.s.s.HttpSolrCall > > > null:org.apache.solr.common.SolrException: Server error writing > document id > > > 6182!bbdbe92468734899c738f048e6f58245 to the index > > > at > > > > org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:240) > > > at > > > > org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67) > > > at > > > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > > at > > > > org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:1002) > > > at > > > > org.apache.solr.update.processor.DistributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:1233) > > > at > > > > org.apache.solr.update.processor.DistributedUpdateProcessor.lambda$versionAdd$2(DistributedUpdateProcessor.java:1082) > > > at > org.apache.solr.update.VersionBucket.runWithLock(VersionBucket.java:50) > > > at > > > > org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1082) > > > at > > > > org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:694) > > > at > > > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > > at > > > > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118) > > > at > > > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > > at > > > > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118) > > > at > > > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > > at > > > > org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:92) > > > at > > > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > > at > > > > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118) > > > at > > > > org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:261) > > > at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:188) > > > at > > > > org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97) > > > at > > > >
Re: Exception from the codec layer during indexing
Hi Rahul, This exception complains that IndexingChain did not deduplicate terms as expected. I don't recall seeing this exception before (which doesn't mean it's not a real bug). What JVM are you running? Does this exception frequently occur or was it a one-off? On Thu, Sep 28, 2023 at 4:49 PM Rahul Goswami wrote: > > Hi, > Following up on my issue...anyone who's seen similar exceptions ? Or any > insights on what might be going on? > > Thanks, > Rahul > > On Wed, Sep 27, 2023 at 1:00 AM Rahul Goswami wrote: > > > Hello, > > On one of the servers running Solr 7.7.2, during indexing I observe 2 > > different kinds of exceptions coming from the Lucene codec layer. I can't > > think of an application/data issue that could be causing this. > > > > In particular, Exception-2 seems like a potential bug since it complains > > about "terms out of order" even though both byte arrays are essentially the > > same. Reason I say this is that the FutureArrays.mismatch() is supposed to > > behave like Java's Arrays.mismatch which returns -1 if NO mismatch is > > found. However the check in the below line treats the value -1 as a > > mismatch causing the exception. > > > > > > https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.7.2/lucene/core/src/java/org/apache/lucene/util/StringHelper.java#L46 > > > > Happy to submit a PR for this if there is a consensus on this being a bug. > > > > Would appreciate any inputs on the exceptions seen below! > > > > *Exception-1:* > > > > 2023-09-19 10:13:48.901 ERROR (qtp1859039536-1691) [ > > x:fsindex_FileIndexer20234799_shard_1] o.a.s.s.HttpSolrCall > > null:org.apache.solr.common.SolrException: Server error writing document id > > 6182!bbdbe92468734899c738f048e6f58245 to the index > > at > > org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:240) > > at > > org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67) > > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:1002) > > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:1233) > > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.lambda$versionAdd$2(DistributedUpdateProcessor.java:1082) > > at org.apache.solr.update.VersionBucket.runWithLock(VersionBucket.java:50) > > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1082) > > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:694) > > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > at > > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118) > > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > at > > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118) > > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > at > > org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:92) > > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > > at > > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118) > > at > > org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:261) > > at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:188) > > at > > org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97) > > at > > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68) > > at > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:202) > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551) > > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711) > > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516) > > at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395) > > at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341) > > at > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) > > at > > com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121) > > at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133) > > at > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) > > at > >
Re: Exception from the codec layer during indexing
Hi, Following up on my issue...anyone who's seen similar exceptions ? Or any insights on what might be going on? Thanks, Rahul On Wed, Sep 27, 2023 at 1:00 AM Rahul Goswami wrote: > Hello, > On one of the servers running Solr 7.7.2, during indexing I observe 2 > different kinds of exceptions coming from the Lucene codec layer. I can't > think of an application/data issue that could be causing this. > > In particular, Exception-2 seems like a potential bug since it complains > about "terms out of order" even though both byte arrays are essentially the > same. Reason I say this is that the FutureArrays.mismatch() is supposed to > behave like Java's Arrays.mismatch which returns -1 if NO mismatch is > found. However the check in the below line treats the value -1 as a > mismatch causing the exception. > > > https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.7.2/lucene/core/src/java/org/apache/lucene/util/StringHelper.java#L46 > > Happy to submit a PR for this if there is a consensus on this being a bug. > > Would appreciate any inputs on the exceptions seen below! > > *Exception-1:* > > 2023-09-19 10:13:48.901 ERROR (qtp1859039536-1691) [ > x:fsindex_FileIndexer20234799_shard_1] o.a.s.s.HttpSolrCall > null:org.apache.solr.common.SolrException: Server error writing document id > 6182!bbdbe92468734899c738f048e6f58245 to the index > at > org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:240) > at > org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67) > at > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:1002) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.doVersionAdd(DistributedUpdateProcessor.java:1233) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.lambda$versionAdd$2(DistributedUpdateProcessor.java:1082) > at org.apache.solr.update.VersionBucket.runWithLock(VersionBucket.java:50) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1082) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:694) > at > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > at > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118) > at > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > at > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118) > at > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > at > org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:92) > at > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) > at > org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118) > at > org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:261) > at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:188) > at > org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97) > at > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:202) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2551) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:516) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341) > at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) > at > com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121) > at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588) > at >
