jtuglu1 commented on code in PR #19532:
URL: https://github.com/apache/druid/pull/19532#discussion_r3597455328
##########
server/src/main/java/org/apache/druid/server/coordinator/loading/SegmentReplicationStatus.java:
##########
@@ -81,4 +96,84 @@ public Map<String, Object2LongMap<String>>
getTierToDatasourceToUnderReplicated(
return tierToUnderReplicated;
}
+
+ /**
+ * Computes unavailable, under-replicated and deep-storage-only segment
counts in a single
+ * pass over {@code usedSegments}, instead of three independent full
iterations. Produces
+ * results identical to calling {@link #getReplicaCountsInCluster} and
+ * {@link #getTierToDatasourceToUnderReplicated} independently for each
segment.
+ *
+ * @param ignoreMissingServers same semantics as in {@link
#getTierToDatasourceToUnderReplicated}.
+ */
+ public SegmentStatsSnapshot computeSegmentStats(Iterable<DataSegment>
usedSegments, boolean ignoreMissingServers)
+ {
+ final Object2IntOpenHashMap<String> datasourceToUnavailable = new
Object2IntOpenHashMap<>();
+ final Object2IntOpenHashMap<String> datasourceToDeepStorageOnly = new
Object2IntOpenHashMap<>();
+ final Map<String, Object2LongMap<String>> tierToUnderReplicated = new
HashMap<>();
+
+ for (DataSegment segment : usedSegments) {
+ final SegmentId segmentId = segment.getId();
+ final String datasource = segment.getDataSource();
+
+ final SegmentReplicaCount totalCount = totalReplicaCounts.get(segmentId);
+ if (totalCount != null && (totalCount.totalLoaded() > 0 ||
totalCount.required() == 0)) {
+ datasourceToUnavailable.addTo(datasource, 0);
+ } else {
+ datasourceToUnavailable.addTo(datasource, 1);
+ }
+ if (totalCount != null && totalCount.totalLoaded() == 0 &&
totalCount.required() == 0) {
+ datasourceToDeepStorageOnly.addTo(datasource, 1);
+ }
+
+ final Map<String, SegmentReplicaCount> tierToReplicaCount =
replicaCountsInTier.get(segmentId);
+ if (tierToReplicaCount != null) {
+ tierToReplicaCount.forEach((tier, counts) -> {
+ final int underReplicated = ignoreMissingServers ? counts.missing()
: counts.missingAndLoadable();
+ if (underReplicated >= 0) {
+ Object2LongOpenHashMap<String> datasourceToUnderReplicated =
(Object2LongOpenHashMap<String>)
+ tierToUnderReplicated.computeIfAbsent(tier, ds -> new
Object2LongOpenHashMap<>());
+ datasourceToUnderReplicated.addTo(datasource, underReplicated);
+ }
+ });
Review Comment:
I can run a few benchmarks and see if this has any material impact (the
original goal was to avoid looping through 3x and instead fuse everything into
a single pass).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
