Tan Lay How created SOLR-8431:
---------------------------------
Summary: Parent shard cannot be deleted after shard splitting
Key: SOLR-8431
URL: https://issues.apache.org/jira/browse/SOLR-8431
Project: Solr
Issue Type: Bug
Affects Versions: 4.10.1
Reporter: Tan Lay How
Priority: Blocker
I performed shard splitting on 2 of the shards out of 3 with the async request
id. The shard splitting task is failed when it trying to attach the replica for
the splitted shard, but i found out on both of the leader splitted shards,
document is splitted correctly (total numFound of both shards = total numFound
of parent shard)
So, i proceed to manually change the clusterstate.json, splitted shards changed
status from construction to active, and parent shards changed from active to
inactive. I also manually attach the replica for the splitted shards, then only
remove the parent shards by unload the parent shards core.
Here the problems come, when i issue a commit in SolrCloud, both of the parent
return back in the solr cloud cloud graph and clusterstate.json with node
status = down & shard status = active.
1) I bring up the parent shards node again and try with another unload core
again, but every time i issue a commit, it will back into the graph and
clusterstate with node status = down & shard status = active.
2) So my second attempt is to delete the shards with collection API using
/admin/collections?action=DELETESHARD
I got the error as below :
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:Cannot
>unload non-existent core
Operation deleteshard caused exception: org.apache.solr.common.SolrException:
Could not fully remove collection: >candidates shard: candidates_shard2 at
org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:364)
at
org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:320)
at
org.apache.solr.handler.admin.CollectionsHandler.handleDeleteShardAction(CollectionsHandler.java:563)
at
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:176)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:729)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:267)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:612)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AutoLoginValve.invoke(AutoLoginValve.java:67) at
org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:304)
at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:82)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683) at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
3) My third attempt is the delete the replica using the collection API
admin/collections?action=DELETEREPLICA
I got the error as below :
org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
Could not remove replica
org.apache.solr.common.SolrException: Could not remove replica :
candidates/candidates_shard2/core_node48 at
org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:364)
at
org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:320)
at
org.apache.solr.handler.admin.CollectionsHandler.handleRemoveReplica(CollectionsHandler.java:495)
at
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:184)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:729)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:267)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:612)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AutoLoginValve.invoke(AutoLoginValve.java:67) at
org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:304)
at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:82)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683) at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
The parent shards cannot be removed and when the delete query came in, Solr
will return "No registered leader was found after waiting xx" due to the parent
shards node status is down but the shards status still appear active in the
cluster.
The same goes for select query, will return error "No server hosting shards".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
