Hi Sushant, while this is true in general, it won't hold here. If you split your index, searching on each splitted shard might be a bit faster, but you'll increase search time much more because Solr needs to send your search queries to all shards and then combine the results. So instead of having one medium fast search request, you'll have several fast requests plus the aggregation step.
Erick is totally right, splitting an index of that size has no performance benefit. Sharding is not a technique to improve performance, it's a technique to be able to handle indexes of hundreds of megabytes in size, which won't fit into an individual machine. Best, Michael Am 20.06.2018 um 19:58 schrieb Sushant Vengurlekar: > Thank you for the detailed response Eric. Very much appreciated. The reason > I am looking into splitting the index into two is because it’s much faster > to search across a smaller index than a larger one. > > On Wed, Jun 20, 2018 at 10:46 AM Erick Erickson <erickerick...@gmail.com> > wrote: > >> You still haven't answered _why_ you think splitting even a 20G index >> is desirable. We regularly see 200G+ indexes per replica in the field, >> so what's the point? Have you measured different setups to see if it's >> a good idea? A 200G index needs some beefy hardware admittedly.... >> >> If you have adequate response times with a 20G index and need to >> increase the QPS rate, just add more replicas. Having more than one >> shard inevitably adds overhead which may (or may not) be made up for >> by parallelizing some of the work. It's nearly always better to use >> only one shard if it meets your response time requirements. >> >> Best, >> Erick >> >> On Wed, Jun 20, 2018 at 10:39 AM, Sushant Vengurlekar >> <svengurle...@curvolabs.com> wrote: >>> The index size is small because this is my local development copy. The >>> production index is more than 20GB. So I am working on getting the index >>> split and replicated on different nodes. Our current instance on prod is >>> single instance solr 6 which we are working on moving towards solrcloud 7 >>> >>> On Wed, Jun 20, 2018 at 10:30 AM Erick Erickson <erickerick...@gmail.com >>> >>> wrote: >>> >>>> Use the indexupgrader tool or optimize your index before using >> splitshard. >>>> >>>> Since this is a small index (< 5G), optimizing will not create an >>>> overly-large segment, so that pitfall is avoided. >>>> >>>> You haven't yet explained why you think splitting the index would be >>>> beneficial. Splitting an index this small is unlikely to improve query >>>> performance appreciably. This feels a lot like an "XY" problem, you're >>>> asking how to do X thinking it will solve Y but not telling us what Y >>>> is. >>>> >>>> Best, >>>> Erick >>>> >>>> On Wed, Jun 20, 2018 at 9:40 AM, Sushant Vengurlekar >>>> <svengurle...@curvolabs.com> wrote: >>>>> How can I resolve this error? >>>>> >>>>> On Wed, Jun 20, 2018 at 9:11 AM, Alexandre Rafalovitch < >>>> arafa...@gmail.com> >>>>> wrote: >>>>> >>>>>> This seems more related to an old index upgraded to latest Solr >> rather >>>> than >>>>>> the split itself. >>>>>> >>>>>> Regards, >>>>>> Alex >>>>>> >>>>>> On Wed, Jun 20, 2018, 12:07 PM Sushant Vengurlekar, < >>>>>> svengurle...@curvolabs.com> wrote: >>>>>> >>>>>>> Thanks for the reply Alessandro! Appreciate it. >>>>>>> >>>>>>> Below is the full request and the error received >>>>>>> >>>>>>> curl ' >>>>>>> >>>>>>> http://localhost:8081/solr/admin/collections?action= >>>>>> SPLITSHARD&collection=dev-transactions&shard=shard1 >>>>>>> ' >>>>>>> >>>>>>> { >>>>>>> >>>>>>> "responseHeader":{ >>>>>>> >>>>>>> "status":500, >>>>>>> >>>>>>> "QTime":7920}, >>>>>>> >>>>>>> "success":{ >>>>>>> >>>>>>> "solr-1:8081_solr":{ >>>>>>> >>>>>>> "responseHeader":{ >>>>>>> >>>>>>> "status":0, >>>>>>> >>>>>>> "QTime":1190}, >>>>>>> >>>>>>> "core":"dev-transactions_shard1_0_replica_n3"}, >>>>>>> >>>>>>> "solr-1:8081_solr":{ >>>>>>> >>>>>>> "responseHeader":{ >>>>>>> >>>>>>> "status":0, >>>>>>> >>>>>>> "QTime":1047}, >>>>>>> >>>>>>> "core":"dev-transactions_shard1_1_replica_n4"}, >>>>>>> >>>>>>> "solr-1:8081_solr":{ >>>>>>> >>>>>>> "responseHeader":{ >>>>>>> >>>>>>> "status":0, >>>>>>> >>>>>>> "QTime":6}}, >>>>>>> >>>>>>> "solr-1:8081_solr":{ >>>>>>> >>>>>>> "responseHeader":{ >>>>>>> >>>>>>> "status":0, >>>>>>> >>>>>>> "QTime":1009}}}, >>>>>>> >>>>>>> "failure":{ >>>>>>> >>>>>>> >>>>>>> >> "solr-1:8081_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$ >>>>>> RemoteSolrException:Error >>>>>>> from server at http://solr-1:8081/solr: >>>>>>> java.lang.IllegalArgumentException: >>>>>>> Cannot merge a segment that has been created with major version 6 >> into >>>>>> this >>>>>>> index which has been created by major version 7"}, >>>>>>> >>>>>>> "Operation splitshard caused >>>>>>> >>>>>>> exception:":"org.apache.solr.common.SolrException:org. >>>>>> apache.solr.common.SolrException: >>>>>>> SPLITSHARD failed to invoke SPLIT core admin command", >>>>>>> >>>>>>> "exception":{ >>>>>>> >>>>>>> "msg":"SPLITSHARD failed to invoke SPLIT core admin command", >>>>>>> >>>>>>> "rspCode":500}, >>>>>>> >>>>>>> "error":{ >>>>>>> >>>>>>> "metadata":[ >>>>>>> >>>>>>> "error-class","org.apache.solr.common.SolrException", >>>>>>> >>>>>>> "root-error-class","org.apache.solr.common.SolrException"], >>>>>>> >>>>>>> "msg":"SPLITSHARD failed to invoke SPLIT core admin command", >>>>>>> >>>>>>> "trace":"org.apache.solr.common.SolrException: SPLITSHARD >> failed >>>> to >>>>>>> invoke SPLIT core admin command\n\tat >>>>>>> >>>>>>> org.apache.solr.client.solrj.SolrResponse.getException( >>>>>> SolrResponse.java:53)\n\tat >>>>>>> >>>>>>> org.apache.solr.handler.admin.CollectionsHandler.invokeAction( >>>>>> CollectionsHandler.java:258)\n\tat >>>>>>> >>>>>>> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody( >>>>>> CollectionsHandler.java:230)\n\tat >>>>>>> >>>>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest( >>>>>> RequestHandlerBase.java:195)\n\tat >>>>>>> >>>>>>> org.apache.solr.servlet.HttpSolrCall.handleAdmin( >>>>>> HttpSolrCall.java:736)\n\tat >>>>>>> >>>>>>> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest( >>>>>> HttpSolrCall.java:717)\n\tat >>>>>>> >> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)\n\tat >>>>>>> >>>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter( >>>>>> SolrDispatchFilter.java:384)\n\tat >>>>>>> >>>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter( >>>>>> SolrDispatchFilter.java:330)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain. >>>>>> doFilter(ServletHandler.java:1629)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle( >>>>>> ServletHandler.java:533)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle( >>>>>> ScopedHandler.java:143)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.security.SecurityHandler.handle( >>>>>> SecurityHandler.java:548)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle( >>>>>> HandlerWrapper.java:132)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.server.handler.ScopedHandler. >>>>>> nextHandle(ScopedHandler.java:190)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.server.session.SessionHandler. >>>>>> doHandle(SessionHandler.java:1595)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.server.handler.ScopedHandler. >>>>>> nextHandle(ScopedHandler.java:188)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.server.handler.ContextHandler. >>>>>> doHandle(ContextHandler.java:1253)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.server.handler.ScopedHandler. >>>>>> nextScope(ScopedHandler.java:168)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.servlet.ServletHandler.doScope( >>>>>> ServletHandler.java:473)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.server.session.SessionHandler. >>>>>> doScope(SessionHandler.java:1564)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.server.handler.ScopedHandler. >>>>>> nextScope(ScopedHandler.java:166)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.server.handler.ContextHandler. >>>>>> doScope(ContextHandler.java:1155)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle( >>>>>> ScopedHandler.java:141)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle( >>>>>> ContextHandlerCollection.java:219)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.server.handler.HandlerCollection. >>>>>> handle(HandlerCollection.java:126)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle( >>>>>> HandlerWrapper.java:132)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle( >>>>>> RewriteHandler.java:335)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle( >>>>>> HandlerWrapper.java:132)\n\tat >>>>>>> org.eclipse.jetty.server.Server.handle(Server.java:530)\n\tat >>>>>>> >>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.server.HttpConnection.onFillable( >>>>>> HttpConnection.java:256)\n\tat >>>>>>> org.eclipse.jetty.io >>>>>>> >>>> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)\ >>>>>> n\tat >>>>>>> org.eclipse.jetty.io >>>> .FillInterest.fillable(FillInterest.java:102)\n\tat >>>>>>> org.eclipse.jetty.io >>>> .ChannelEndPoint$2.run(ChannelEndPoint.java:124)\n\ >>>>>> tat >>>>>>> >>>>>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill. >>>>>> doProduce(EatWhatYouKill.java:247)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill. >>>>>> produce(EatWhatYouKill.java:140)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run( >>>>>> EatWhatYouKill.java:131)\n\tat >>>>>>> >>>>>>> >>>> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run( >>>>>> ReservedThreadExecutor.java:382)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob( >>>>>> QueuedThreadPool.java:708)\n\tat >>>>>>> >>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run( >>>>>> QueuedThreadPool.java:626)\n\tat >>>>>>> java.lang.Thread.run(Thread.java:748)\n", >>>>>>> >>>>>>> "code":500}} >>>>>>> >>>>>>> >>>>>>> On Wed, Jun 20, 2018 at 2:25 AM, Alessandro Benedetti < >>>>>>> a.benede...@sease.io> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> in the first place, why do you want to split 2 Gb indexes ? >>>>>>>> Nowadays is a fairly small index. >>>>>>>> >>>>>>>> Secondly what you reported is incomplete. >>>>>>>> I would expect a Caused By section in the stacktrace. >>>>>>>> >>>>>>>> This are generic recommendations, always spend time in analysing >> the >>>>>>>> problem >>>>>>>> you had scrupulously. >>>>>>>> - SolrCloud problems often involve more than one node. Be sure to >>>> check >>>>>>> the >>>>>>>> logs of all the nodes possibly involved. >>>>>>>> - Report the full stack trace to the community >>>>>>>> - Report your full request which provoked the exception >>>>>>>> >>>>>>>> Help is much easier this way :) >>>>>>>> >>>>>>>> Regards >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ----- >>>>>>>> --------------- >>>>>>>> Alessandro Benedetti >>>>>>>> Search Consultant, R&D Software Engineer, Director >>>>>>>> Sease Ltd. - www.sease.io >>>>>>>> -- >>>>>>>> Sent from: >>>> http://lucene.472066.n3.nabble.com/Solr-User-f472068.html >>>>>>>> >>>>>>> >>>>>> >>>> >> >
