[ https://issues.apache.org/jira/browse/SOLR-14909 ]
Alexey Bulygin deleted comment on SOLR-14909:
---------------------------------------
was (Author: [email protected]):
I look AddReplicaCmd source, please see my patch.
The goal is to skip the Assignment step when adding replicas( in my case
offline indexing takes place - i copy the indexes on sftp, then add a replica),
that is, I need skipNodeAssignment = Boolean.getBoolean
("solr.custom.assign.replicas. positions.skip ") by default, for example.
Then, I skip build assignment step, but do not skip the creation of the state
in the zk in later:
if (! skipCreateReplicaInClusterState ||! skipNodeAssignment) {
positions = Assign.getNodesForNewReplicas (clusterState,
collection.getName (), sliceName, numNrtReplicas,
- numTlogReplicas, numPullReplicas, createNodeSetStr, cloudManager);
Please tell me if it looks good?
> Add replica is very slow on a large cluster
> -------------------------------------------
>
> Key: SOLR-14909
> URL: https://issues.apache.org/jira/browse/SOLR-14909
> Project: Solr
> Issue Type: Task
> Affects Versions: 7.6
> Reporter: Varun Thacker
> Priority: Major
> Attachments: skipAssignBuildReplicaPos.patch
>
>
> We create ~100 collections every day for new incoming data
> We first issue a create-collection request for all the collections (4 shards
> and createNodeSet=empty). This would create collections with no replicas
> We then issue async add-replica calls for all the shards creating 1 replica
> each. 100 collection X 4 shards = 400 add-replica calls. All the add replica
> calls pass the node parameter telling Solr where the replica should be created
> The cluster has 190 nodes currently and when we upgraded to Solr 7.7.3 we
> noticed that the add replicas took 2 hours and 45 mins to complete! Clearly
> something was wrong as the same cluster previously running Solr 7.3.1 was
> taking a few mins only.
> A thread dump of the overseer showed a 100 threads stuck here ( Why 100?
> That's the Solr default thread pool size set by MAX_PARALLEL_TASKS in
> OverseerTaskProcessor )
>
> {code:java}
> "OverseerThreadFactory-13-thread-1226-processing-n:10.128.18.69:8983_solr"
> #11163 prio=5 os_prio=0 cpu=0.69ms elapsed=987.97s tid=0x00007f01f8051000
> nid=0xd7a waiting for monitor entry [0x00007f01c1121000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Object.wait([email protected]/Native Method)
> - waiting on <no object reference available>
> at
> org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper$SessionRef.get(PolicyHelper.java:449)
> - waiting to re-lock in wait() <0x00000007259e6a98> (a java.lang.Object)
> at
> org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper.getSession(PolicyHelper.java:493)
> at
> org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper.getReplicaLocations(PolicyHelper.java:121)
> at
> org.apache.solr.cloud.api.collections.Assign.getPositionsUsingPolicy(Assign.java:382)
> at
> org.apache.solr.cloud.api.collections.Assign$PolicyBasedAssignStrategy.assign(Assign.java:630)
> at
> org.apache.solr.cloud.api.collections.Assign.getNodesForNewReplicas(Assign.java:368)
> at
> org.apache.solr.cloud.api.collections.AddReplicaCmd.buildReplicaPositions(AddReplicaCmd.java:360)
> at
> org.apache.solr.cloud.api.collections.AddReplicaCmd.addReplica(AddReplicaCmd.java:146)
> at
> org.apache.solr.cloud.api.collections.AddReplicaCmd.call(AddReplicaCmd.java:91)
> at
> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:294)
> {code}
>
>
> It's strange because each add-replica API call would create a single replica
> and specify which node is must be created on.
>
> Assign.getNodesForNewReplicas is where the slowdown was and we noticed
> SKIP_NODE_ASSIGNMENT flag (
> https://github.com/apache/lucene-solr/commit/17cb1b17172926d0d9aed3dfd3b9adb90cf65e0f#diff-ee29887eff6e474e58fcf3c02077f179R355
> ) that the overseer reads could have skipped the method from being called.
> So we started passing SKIP_NODE_ASSIGNMENT=true and still no luck! The
> replicas took just as long to create. It turned out that the Collections
> Handler wasn't passing the SKIP_NODE_ASSIGNMENT parameter to the overseer.
> The add replica call only passes a specific set of params to the overseer
> https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.7.3/solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java#L823
> . We changed this to also pass SKIP_NODE_ASSIGNMENT.
> Now when we try to create the replicas it takes 4 minutes approximately vs 2
> hours 45 mins that it was taking previosuly.
> Only master respects that param to the overseer (
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java#L938
> ) . However it doesn't matter in master because the autoscaling framework is
> gone ( https://github.com/apache/lucene-solr/commit/cc0c111/ )
> I believe this will be seen in all versions since Solr 7.6 (
> https://issues.apache.org/jira/browse/SOLR-12739 ) through every 8.x release
> Lastly, I manually tried to add a replica with and without the flag. Without
> the flag it took 20 second and with the flag 2 seconds.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
