sthetland commented on a change in pull request #9965:
URL: https://github.com/apache/druid/pull/9965#discussion_r438362630
##########
File path: docs/operations/api-reference.md
##########
@@ -114,6 +114,41 @@ Returns the number of segments to load and drop, as well
as the total segment lo
Returns the serialized JSON of segments to load and drop for each Historical
process.
+
+#### Segment Loading for Datasource
Review comment:
```suggestion
#### Segment Loading by Datasource
```
##########
File path: docs/operations/api-reference.md
##########
@@ -114,6 +114,41 @@ Returns the number of segments to load and drop, as well
as the total segment lo
Returns the serialized JSON of segments to load and drop for each Historical
process.
+
+#### Segment Loading for Datasource
+
+These APIs can be used to verify if segments created by recent ingestion task
are loaded onto historicals and available for query.
+An example workflow for this is:
+1. Submit your ingestion task
+2. Repeatedly poll Overlord's task API (
`/druid/indexer/v1/task/{taskId}/status`) until task is completed and succeeded.
+3. Poll Segment Loading for Datasource API
(`/druid/coordinator/v1/datasources/{dataSourceName}/loadstatus`) with
`forceMetadataRefresh=true` once.
+If there are segments not yet loaded, continue to step 4, otherwise you can
now query the data.
+4. Repeatedly poll Segment Loading for Datasource API
(`/druid/coordinator/v1/datasources/{dataSourceName}/loadstatus`) with
`forceMetadataRefresh=false`.
+Continue polling until all segments are loaded. Once all segments are loaded
you can now query the data.
+
+##### GET
+
+*
`/druid/coordinator/v1/datasources/{dataSourceName}/loadstatus?forceMetadataRefresh={boolean}&interval={myInterval}`
+
+Returns the percentage of segments actually loaded in the cluster versus
segments that should be loaded in the cluster for the given datasource
+over the given interval (or last 2 weeks if interval is not given). Setting
`forceMetadataRefresh=true`
Review comment:
Maybe indicate that true is the default here? and remove the sentence
that comes later ("`forceMetadataRefresh` will be set to true if not given.")
```suggestion
over the given interval (or last 2 weeks if interval is not given). Setting
`forceMetadataRefresh` to true (the default)
```
##########
File path: docs/operations/api-reference.md
##########
@@ -114,6 +114,41 @@ Returns the number of segments to load and drop, as well
as the total segment lo
Returns the serialized JSON of segments to load and drop for each Historical
process.
+
+#### Segment Loading for Datasource
+
+These APIs can be used to verify if segments created by recent ingestion task
are loaded onto historicals and available for query.
Review comment:
```suggestion
You can verify if segments created by a recent ingestion task are loaded
onto historicals and available for querying using the following APIs.
```
##########
File path: docs/operations/api-reference.md
##########
@@ -114,6 +114,41 @@ Returns the number of segments to load and drop, as well
as the total segment lo
Returns the serialized JSON of segments to load and drop for each Historical
process.
+
+#### Segment Loading for Datasource
+
+These APIs can be used to verify if segments created by recent ingestion task
are loaded onto historicals and available for query.
+An example workflow for this is:
+1. Submit your ingestion task
Review comment:
```suggestion
1. Submit your ingestion task.
```
##########
File path: docs/operations/api-reference.md
##########
@@ -114,6 +114,41 @@ Returns the number of segments to load and drop, as well
as the total segment lo
Returns the serialized JSON of segments to load and drop for each Historical
process.
+
+#### Segment Loading for Datasource
+
+These APIs can be used to verify if segments created by recent ingestion task
are loaded onto historicals and available for query.
+An example workflow for this is:
+1. Submit your ingestion task
+2. Repeatedly poll Overlord's task API (
`/druid/indexer/v1/task/{taskId}/status`) until task is completed and succeeded.
+3. Poll Segment Loading for Datasource API
(`/druid/coordinator/v1/datasources/{dataSourceName}/loadstatus`) with
`forceMetadataRefresh=true` once.
+If there are segments not yet loaded, continue to step 4, otherwise you can
now query the data.
+4. Repeatedly poll Segment Loading for Datasource API
(`/druid/coordinator/v1/datasources/{dataSourceName}/loadstatus`) with
`forceMetadataRefresh=false`.
+Continue polling until all segments are loaded. Once all segments are loaded
you can now query the data.
+
+##### GET
+
+*
`/druid/coordinator/v1/datasources/{dataSourceName}/loadstatus?forceMetadataRefresh={boolean}&interval={myInterval}`
+
+Returns the percentage of segments actually loaded in the cluster versus
segments that should be loaded in the cluster for the given datasource
+over the given interval (or last 2 weeks if interval is not given). Setting
`forceMetadataRefresh=true`
+will force the coordinator to poll latest segment metadata from the metadata
store. `forceMetadataRefresh` will be set to true if not given.
+If no used segments found for the given inputs, this API returns 100% as the
value.
+
+ *
`/druid/coordinator/v1/datasources/{dataSourceName}/loadstatus?simple&forceMetadataRefresh={boolean}&interval={myInterval}`
+
+Returns the number of segments left to load until segments that should be
loaded in the cluster are available for the given datasource
+over the given interval (or last 2 weeks if interval is not given). This does
not include replication. Setting `forceMetadataRefresh=true`
Review comment:
"This does not include replication" Meaning, replicated segments?
##########
File path: docs/operations/api-reference.md
##########
@@ -114,6 +114,41 @@ Returns the number of segments to load and drop, as well
as the total segment lo
Returns the serialized JSON of segments to load and drop for each Historical
process.
+
+#### Segment Loading for Datasource
+
+These APIs can be used to verify if segments created by recent ingestion task
are loaded onto historicals and available for query.
+An example workflow for this is:
+1. Submit your ingestion task
+2. Repeatedly poll Overlord's task API (
`/druid/indexer/v1/task/{taskId}/status`) until task is completed and succeeded.
+3. Poll Segment Loading for Datasource API
(`/druid/coordinator/v1/datasources/{dataSourceName}/loadstatus`) with
`forceMetadataRefresh=true` once.
Review comment:
maybe call it loadstatus?
```suggestion
3. Poll the datasource loadstatus API
(`/druid/coordinator/v1/datasources/{dataSourceName}/loadstatus`) with
`forceMetadataRefresh=true` once.
```
##########
File path: docs/operations/api-reference.md
##########
@@ -114,6 +114,41 @@ Returns the number of segments to load and drop, as well
as the total segment lo
Returns the serialized JSON of segments to load and drop for each Historical
process.
+
+#### Segment Loading for Datasource
+
+These APIs can be used to verify if segments created by recent ingestion task
are loaded onto historicals and available for query.
+An example workflow for this is:
+1. Submit your ingestion task
+2. Repeatedly poll Overlord's task API (
`/druid/indexer/v1/task/{taskId}/status`) until task is completed and succeeded.
+3. Poll Segment Loading for Datasource API
(`/druid/coordinator/v1/datasources/{dataSourceName}/loadstatus`) with
`forceMetadataRefresh=true` once.
+If there are segments not yet loaded, continue to step 4, otherwise you can
now query the data.
+4. Repeatedly poll Segment Loading for Datasource API
(`/druid/coordinator/v1/datasources/{dataSourceName}/loadstatus`) with
`forceMetadataRefresh=false`.
+Continue polling until all segments are loaded. Once all segments are loaded
you can now query the data.
+
+##### GET
+
+*
`/druid/coordinator/v1/datasources/{dataSourceName}/loadstatus?forceMetadataRefresh={boolean}&interval={myInterval}`
+
+Returns the percentage of segments actually loaded in the cluster versus
segments that should be loaded in the cluster for the given datasource
+over the given interval (or last 2 weeks if interval is not given). Setting
`forceMetadataRefresh=true`
+will force the coordinator to poll latest segment metadata from the metadata
store. `forceMetadataRefresh` will be set to true if not given.
+If no used segments found for the given inputs, this API returns 100% as the
value.
Review comment:
Maybe add the positive case as well, if this is the case?
```suggestion
If all segments have been loaded or no used segments are found for the given
inputs, this API returns 100% as the value.
```
##########
File path: docs/operations/api-reference.md
##########
@@ -114,6 +114,41 @@ Returns the number of segments to load and drop, as well
as the total segment lo
Returns the serialized JSON of segments to load and drop for each Historical
process.
+
+#### Segment Loading for Datasource
+
+These APIs can be used to verify if segments created by recent ingestion task
are loaded onto historicals and available for query.
+An example workflow for this is:
+1. Submit your ingestion task
+2. Repeatedly poll Overlord's task API (
`/druid/indexer/v1/task/{taskId}/status`) until task is completed and succeeded.
Review comment:
```suggestion
2. Repeatedly poll the Overlord's task API (
`/druid/indexer/v1/task/{taskId}/status`) until your task is shown to be
successfully completed.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
