https://issues.apache.org/jira/browse/SOLR-16588
already filed and fixed. Kevin Risden On Sat, Jan 14, 2023 at 7:11 AM Dmitry Grechka <[email protected]> wrote: > Hi all, > > It seems that I found an issue in Solr 9.1. > > After upgrading from solr 9.0 to solr 9.1 (solr cloud, deployed with > official docker images) I started to get cores failing during indexing > of dense vector with the following error > > { > > "responseHeader":{ > > "rf":2147483647, > > "status":500, > > "QTime":4135}, > > "error":{ > > "metadata":[ > > "error-class","org.apache.solr.common.SolrException", > > "root-error-class","org.apache.solr.common.SolrException", > > > "error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException", > > > "root-error-class","org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException"], > > "msg":"Async exception during distributed update: Error from server > at http://10.42.4.75:8983/solr/kashtankaimages_shard7_replica_n24/: > null\n\n\n\nrequest: > > http://10.42.4.75:8983/solr/kashtankaimages_shard7_replica_n24/\nRemote > < > http://10.42.4.75:8983/solr/kashtankaimages_shard7_replica_n24/nRemote> > error message: Server error writing document id > vk-poterjashkansk/1675252/1 to the index", > > > "trace":"org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException: > Async exception during distributed update: Error from server at > http://10.42.4.75:8983/solr/kashtankaimages_shard7_replica_n24/: > null\n\n\n\nrequest: > > http://10.42.4.75:8983/solr/kashtankaimages_shard7_replica_n24/\nRemote > < > http://10.42.4.75:8983/solr/kashtankaimages_shard7_replica_n24/nRemote> > error message: Server error writing document id > vk-poterjashkansk/1675252/1 to the index\n\tat > > org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1420)\n\tat > > org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1253)\n\tat > > org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:197)\n\tat > > org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:78)\n\tat > > org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:78)\n\tat > > org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:78)\n\tat > > org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:78)\n\tat > > org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:78)\n\tat > > org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:78)\n\tat > > org.apache.solr.update.processor.UpdateRequestProcessor.finish(UpdateRequestProcessor.java:78)\n\tat > > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:94)\n\tat > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)\n\tat > org.apache.solr.core.SolrCore.execute(SolrCore.java:2865)\n\tat > > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:887)\n\tat > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:606)\n\tat > > org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:250)\n\tat > > org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:218)\n\tat > > org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257)\n\tat > > org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227)\n\tat > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:213)\n\tat > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)\n\tat > > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)\n\tat > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)\n\tat > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)\n\tat > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat > > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)\n\tat > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)\n\tat > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)\n\tat > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)\n\tat > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)\n\tat > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)\n\tat > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)\n\tat > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat > > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)\n\tat > > org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)\n\tat > > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)\n\tat > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat > > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)\n\tat > > org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)\n\tat > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat > org.eclipse.jetty.server.Server.handle(Server.java:516)\n\tat > > org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)\n\tat > > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)\n\tat > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)\n\tat > > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)\n\tat > org.eclipse.jetty.io > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat > org.eclipse.jetty.io > .FillInterest.fillable(FillInterest.java:105)\n\tat > org.eclipse.jetty.io > .ChannelEndPoint$1.run(ChannelEndPoint.java:104)\n\tat > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)\n\tat > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)\n\tat > java.base/java.lang.Thread.run(Unknown Source)\n", > > "code":500}} > > > and in logs > > > 2023-01-14 05:51:15.007 ERROR (qtp35984028-128) [c:kashtankaimages > s:shard8 r:core_node100 x:kashtankaimages_shard8_replica_n99] > o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: > this IndexWriter is closed => org.apache.solr.common.SolrException: > this IndexWriter is closed > > at > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:243) > > org.apache.solr.common.SolrException: this IndexWriter is closed > > at > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:243) > ~[?:?] > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2865) ~[?:?] > > at > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:887) > ~[?:?] > > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:606) > ~[?:?] > > at > > org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:250) > ~[?:?] > > at > > org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:218) > ~[?:?] > > at > > org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257) > ~[?:?] > > at > > org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227) > ~[?:?] > > at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:213) > ~[?:?] > > at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195) > ~[?:?] > > at > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201) > ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) > ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) > ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) > ~[jetty-security-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) > ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322) > ~[jetty-rewrite-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at org.eclipse.jetty.server.Server.handle(Server.java:516) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:439) > ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137) > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:183) > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:138) > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:361) > ~[http2-common-9.4.48.v20220622.jar:9.4.48.v20220622] > > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) > ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) > ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at > > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) > ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622] > > at java.lang.Thread.run(Unknown Source) ~[?:?] > > Caused by: org.apache.lucene.store.AlreadyClosedException: this > IndexWriter is closed > > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:908) > ~[?:?] > > at > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:921) > ~[?:?] > > at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3832) > ~[?:?] > > at > > org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:749) > ~[?:?] > > at > > org.apache.solr.update.processor.RunUpdateProcessorFactory$RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:100) > ~[?:?] > > at > > org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:67) > ~[?:?] > > at > > org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1231) > ~[?:?] > > at > > org.apache.solr.update.processor.DistributedZkUpdateProcessor.processCommit(DistributedZkUpdateProcessor.java:250) > ~[?:?] > > at > > org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:173) > ~[?:?] > > at > > org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:68) > ~[?:?] > > at > > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:77) > ~[?:?] > > at > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224) > ~[?:?] > > ... 54 more > > *Caused by: org.apache.solr.common.SolrException: null KNN algorithm > is not supported* > > at > > org.apache.solr.core.SchemaCodecFactory$1.getKnnVectorsFormatForField(SchemaCodecFactory.java:134) > ~[?:?] > > at > > org.apache.lucene.codecs.lucene92.Lucene92Codec$3.getKnnVectorsFormatForField(Lucene92Codec.java:101) > ~[?:?] > > at > > org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.getInstance(PerFieldKnnVectorsFormat.java:148) > ~[?:?] > > at > > org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.writeField(PerFieldKnnVectorsFormat.java:107) > ~[?:?] > > at > > org.apache.lucene.index.VectorValuesWriter.flush(VectorValuesWriter.java:145) > ~[?:?] > > at > > org.apache.lucene.index.IndexingChain.writeVectors(IndexingChain.java:461) > ~[?:?] > > at > org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:265) > ~[?:?] > > at > > org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:391) > ~[?:?] > > at > > org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:493) > ~[?:?] > > at > > org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:672) > ~[?:?] > > at > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:569) > ~[?:?] > > at > > org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381) > ~[?:?] > > at > > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:366) > ~[?:?] > > at > > org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:246) > ~[?:?] > > at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2305) > ~[?:?] > > at > > org.apache.solr.update.UpdateLog.openRealtimeSearcher(UpdateLog.java:743) > ~[?:?] > > at > > org.apache.solr.update.IndexFingerprint.getFingerprint(IndexFingerprint.java:95) > ~[?:?] > > at > > org.apache.solr.handler.component.RealTimeGetComponent.processGetFingeprint(RealTimeGetComponent.java:1200) > ~[?:?] > > at > > org.apache.solr.handler.component.RealTimeGetComponent.process(RealTimeGetComponent.java:153) > ~[?:?] > > at > > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:420) > ~[?:?] > > at > > org.apache.solr.handler.RealTimeGetHandler.handleRequestBody(RealTimeGetHandler.java:44) > ~[?:?] > > at > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224) > ~[?:?] > > > I have the following schema > > <fieldType name="feature_vector_1024" class="solr.DenseVectorField" > vectorDimension="1024" similarityFunction="dot_product"/> > > <field name="calvin_zhirui_embedding" type="feature_vector_1024" > indexed="true" stored="true"/> > > > Note, that "knnAlgorithm" is not specified. > > Everything worked well in Solr 9.0, the issue appeared after switching > to the Docker image solr:9.1 > > > > When I updated the schema to explicitly state default values, the issue > has gone. Thus the following schema works in solr:9.1 > > <fieldType name="feature_vector_1024" class="solr.DenseVectorField" > vectorDimension="1024" similarityFunction="dot_product" > knnAlgorithm="hnsw" hnswMaxConnections="16" hnswBeamWidth="100"/> > > > > > I can define the issue as "default value of knnAlgorithm property of > solr.DenseVectorField is not applied." > > > Shall I post the issue to Jira? > > > Kind regards, > > Dmitry. > >
