clintropolis commented on a change in pull request #9965:
URL: https://github.com/apache/druid/pull/9965#discussion_r440497521
##########
File path: docs/operations/api-reference.md
##########
@@ -114,6 +114,44 @@ Returns the number of segments to load and drop, as well
as the total segment lo
Returns the serialized JSON of segments to load and drop for each Historical
process.
+
+#### Segment Loading by Datasource
+
+You can verify if segments created by a recent ingestion task are loaded onto
historicals and available for querying using the following APIs.
+An example workflow for this is:
Review comment:
This example workflow seems useful, but also out of place given the rest
of this document, which strictly documents API requests and responses instead
of illustrating an example application of how to use the API. I'm not sure
where else it should live, or if it should just be removed, or if maybe it is
fine, but I think it is worth discussing.
##########
File path: docs/operations/api-reference.md
##########
@@ -96,11 +96,11 @@ Returns the percentage of segments actually loaded in the
cluster versus segment
* `/druid/coordinator/v1/loadstatus?simple`
-Returns the number of segments left to load until segments that should be
loaded in the cluster are available for queries. This does not include
replication.
+Returns the number of segments left to load until segments that should be
loaded in the cluster are available for queries. This does not include replica
of segments.
* `/druid/coordinator/v1/loadstatus?full`
-Returns the number of segments left to load in each tier until segments that
should be loaded in the cluster are all available. This includes replication.
+Returns the number of segments left to load in each tier until segments that
should be loaded in the cluster are all available. This include replica of
segments.
Review comment:
"This include replica of segments." doesn't quite seem right, maybe
"This includes segment replication counts."
##########
File path:
server/src/main/java/org/apache/druid/server/coordinator/DruidCoordinator.java
##########
@@ -256,15 +256,24 @@ public boolean isLeader()
* @return tier -> { dataSource -> underReplicationCount } map
*/
public Map<String, Object2LongMap<String>>
computeUnderReplicationCountsPerDataSourcePerTier()
+ {
+ final Iterable<DataSegment> dataSegments =
segmentsMetadataManager.iterateAllUsedSegments();
+ return
computeUnderReplicationCountsPerDataSourcePerTierForSegments(dataSegments);
+ }
+
+ /**
+ * @return tier -> { dataSource -> underReplicationCount } map
+ */
+ public Map<String, Object2LongMap<String>>
computeUnderReplicationCountsPerDataSourcePerTierForSegments(
Review comment:
I think it's ok that `segmentReplicantLookup` could potentially be stale
since it is only updated on coordinator runs, since the refreshed segment
metadata would at least ensure that the stale data would be under counting
replication levels, rather than potentially falsely reporting that everything
is available (when forcing refresh).
##########
File path: docs/operations/api-reference.md
##########
@@ -96,11 +96,11 @@ Returns the percentage of segments actually loaded in the
cluster versus segment
* `/druid/coordinator/v1/loadstatus?simple`
-Returns the number of segments left to load until segments that should be
loaded in the cluster are available for queries. This does not include
replication.
+Returns the number of segments left to load until segments that should be
loaded in the cluster are available for queries. This does not include replica
of segments.
Review comment:
"This does not include replica of segments." -> "This does not include
segment replication counts."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
