psalagnac commented on code in PR #5114: URL: https://github.com/apache/solr/pull/5114#discussion_r4287897139
########## solr/core/src/test/org/apache/solr/core/snapshots/TestSolrCloudSnapshotInactiveShard.java: ########## @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.solr.core.snapshots; + +import java.util.Collection; +import org.apache.lucene.tests.util.LuceneTestCase; +import org.apache.solr.client.solrj.impl.CloudSolrClient; +import org.apache.solr.client.solrj.request.CollectionAdminRequest; +import org.apache.solr.cloud.SolrCloudTestCase; +import org.apache.solr.common.SolrException; +import org.apache.solr.common.cloud.Replica; +import org.apache.solr.common.cloud.ZkStateReader; +import org.apache.solr.embedded.JettySolrRunner; +import org.junit.BeforeClass; +import org.junit.Test; + +/** + * A collection snapshot must not be recorded as successful when a shard has no active replica to + * take a core snapshot. + */ [email protected]({"SimpleText"}) Review Comment: Is there any good reason for codec suppression? This test does not seem to depend on it. ########## solr/core/src/java/org/apache/solr/cloud/api/collections/CreateSnapshotCmd.java: ########## @@ -166,26 +166,20 @@ public void call(AdminCmdContext adminCmdContext, ZkNodeProps message, NamedList "Unable to create a snapshot with name {} for following cores {}", commitName, shardByCoreName.keySet()); + } - // Count number of failures per shard. - Map<String, Integer> failuresByShardId = new HashMap<>(); - for (Map.Entry<String, Slice> entry : shardByCoreName.entrySet()) { - int f = 0; - if (failuresByShardId.get(entry.getValue().getName()) != null) { - f = failuresByShardId.get(entry.getValue().getName()); - } - failuresByShardId.put(entry.getValue().getName(), f + 1); - } - - // Now that we know number of failures per shard, we can figure out - // if at-least one replica per shard was able to create a snapshot or not. - DocCollection collectionStatus = - ccc.getZkStateReader().getClusterState().getCollection(collectionName); - for (Map.Entry<String, Integer> entry : failuresByShardId.entrySet()) { - int replicaCount = collectionStatus.getSlice(entry.getKey()).getReplicas().size(); - if (replicaCount <= entry.getValue()) { - failedShards.add(entry.getKey()); - } + // A shard counts as captured only if at least one of its replicas reported a snapshot. Shards + // whose replicas were all skipped as inactive never received a request, so they must be + // checked against the cluster state rather than against the requests that were sent. + Set<String> shardsWithSnapshot = new HashSet<>(); + for (CoreSnapshotMetaData replica : replicas) { + shardsWithSnapshot.add(replica.getShardId()); + } + DocCollection collectionStatus = + ccc.getZkStateReader().getClusterState().getCollection(collectionName); + for (Slice slice : collectionStatus.getActiveSlices()) { Review Comment: We iterate only over **active** slices, while the initial loop to get all replicas (near line 102) considers all slices, not just active ones. This seems inconsistent. If we don't consider the non active shards as part of the success/failure criteria, we could skip them initially. ########## solr/core/src/java/org/apache/solr/cloud/api/collections/CreateSnapshotCmd.java: ########## @@ -166,26 +166,20 @@ public void call(AdminCmdContext adminCmdContext, ZkNodeProps message, NamedList "Unable to create a snapshot with name {} for following cores {}", commitName, shardByCoreName.keySet()); + } - // Count number of failures per shard. - Map<String, Integer> failuresByShardId = new HashMap<>(); - for (Map.Entry<String, Slice> entry : shardByCoreName.entrySet()) { - int f = 0; - if (failuresByShardId.get(entry.getValue().getName()) != null) { - f = failuresByShardId.get(entry.getValue().getName()); - } - failuresByShardId.put(entry.getValue().getName(), f + 1); - } - - // Now that we know number of failures per shard, we can figure out - // if at-least one replica per shard was able to create a snapshot or not. - DocCollection collectionStatus = - ccc.getZkStateReader().getClusterState().getCollection(collectionName); - for (Map.Entry<String, Integer> entry : failuresByShardId.entrySet()) { - int replicaCount = collectionStatus.getSlice(entry.getKey()).getReplicas().size(); - if (replicaCount <= entry.getValue()) { - failedShards.add(entry.getKey()); - } + // A shard counts as captured only if at least one of its replicas reported a snapshot. Shards + // whose replicas were all skipped as inactive never received a request, so they must be + // checked against the cluster state rather than against the requests that were sent. + Set<String> shardsWithSnapshot = new HashSet<>(); + for (CoreSnapshotMetaData replica : replicas) { + shardsWithSnapshot.add(replica.getShardId()); + } + DocCollection collectionStatus = + ccc.getZkStateReader().getClusterState().getCollection(collectionName); Review Comment: I'd rather we don't read the collection state a second time. We should use same version we previously read. Second call is done after `processResponses()`, which is blocking and can take a while. The collection state might have slightly changed. The old code had the same double read, but it only ran on the failure path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
