kfaraz commented on code in PR #16616:
URL: https://github.com/apache/druid/pull/16616#discussion_r1643872527
##########
docs/operations/metrics.md:
##########
@@ -264,40 +264,42 @@ If the JVM does not support CPU time measurement for the
current thread, `ingest
## Indexing service
-|Metric|Description|Dimensions|Normal value|
-|------|-----------|----------|------------|
-|`task/run/time`|Milliseconds taken to run a task.| `dataSource`, `taskId`,
`taskType`, `groupId`, `taskStatus`, `tags`|Varies|
-|`task/pending/time`|Milliseconds taken for a task to wait for running.|
`dataSource`, `taskId`, `taskType`, `groupId`, `tags`|Varies|
-|`task/action/log/time`|Milliseconds taken to log a task action to the audit
log.| `dataSource`, `taskId`, `taskType`, `groupId`, `taskActionType`, `tags`|<
1000 (subsecond)|
-|`task/action/run/time`|Milliseconds taken to execute a task action.|
`dataSource`, `taskId`, `taskType`, `groupId`, `taskActionType`, `tags`|Varies
from subsecond to a few seconds, based on action type.|
-|`task/action/success/count`|Number of task actions that were executed
successfully during the emission period. Currently only being emitted for
[batched `segmentAllocate`
actions](../ingestion/tasks.md#batching-segmentallocate-actions).|
`dataSource`, `taskId`, `taskType`, `groupId`, `taskActionType`, `tags`|Varies|
-|`task/action/failed/count`|Number of task actions that failed during the
emission period. Currently only being emitted for [batched `segmentAllocate`
actions](../ingestion/tasks.md#batching-segmentallocate-actions).|
`dataSource`, `taskId`, `taskType`, `groupId`, `taskActionType`, `tags`|Varies|
-|`task/action/batch/queueTime`|Milliseconds spent by a batch of task actions
in queue. Currently only being emitted for [batched `segmentAllocate`
actions](../ingestion/tasks.md#batching-segmentallocate-actions).|
`dataSource`, `taskActionType`, `interval`|Varies based on the
`batchAllocationWaitTime` and number of batches in queue.|
-|`task/action/batch/runTime`|Milliseconds taken to execute a batch of task
actions. Currently only being emitted for [batched `segmentAllocate`
actions](../ingestion/tasks.md#batching-segmentallocate-actions).|
`dataSource`, `taskActionType`, `interval`|Varies from subsecond to a few
seconds, based on action type and batch size.|
-|`task/action/batch/size`|Number of task actions in a batch that was executed
during the emission period. Currently only being emitted for [batched
`segmentAllocate`
actions](../ingestion/tasks.md#batching-segmentallocate-actions).|
`dataSource`, `taskActionType`, `interval`|Varies based on number of concurrent
task actions.|
-|`task/action/batch/attempts`|Number of execution attempts for a single batch
of task actions. Currently only being emitted for [batched `segmentAllocate`
actions](../ingestion/tasks.md#batching-segmentallocate-actions).|
`dataSource`, `taskActionType`, `interval`|1 if there are no failures or
retries.|
-|`task/segmentAvailability/wait/time`|The amount of milliseconds a batch
indexing task waited for newly created segments to become available for
querying.| `dataSource`, `taskType`, `groupId`, `taskId`,
`segmentAvailabilityConfirmed`, `tags`|Varies|
-|`segment/added/bytes`|Size in bytes of new segments created.| `dataSource`,
`taskId`, `taskType`, `groupId`, `interval`, `tags`|Varies|
-|`segment/moved/bytes`|Size in bytes of segments moved/archived via the Move
Task.| `dataSource`, `taskId`, `taskType`, `groupId`, `interval`, `tags`|Varies|
-|`segment/nuked/bytes`|Size in bytes of segments deleted via the Kill Task.|
`dataSource`, `taskId`, `taskType`, `groupId`, `interval`, `tags`|Varies|
-|`task/success/count`|Number of successful tasks per emission period. This
metric is only available if the `TaskCountStatsMonitor` module is included.|
`dataSource`|Varies|
-|`task/failed/count`|Number of failed tasks per emission period. This metric
is only available if the `TaskCountStatsMonitor` module is
included.|`dataSource`|Varies|
-|`task/running/count`|Number of current running tasks. This metric is only
available if the `TaskCountStatsMonitor` module is
included.|`dataSource`|Varies|
-|`task/pending/count`|Number of current pending tasks. This metric is only
available if the `TaskCountStatsMonitor` module is
included.|`dataSource`|Varies|
-|`task/waiting/count`|Number of current waiting tasks. This metric is only
available if the `TaskCountStatsMonitor` module is
included.|`dataSource`|Varies|
-|`taskSlot/total/count`|Number of total task slots per emission period. This
metric is only available if the `TaskSlotCountStatsMonitor` module is
included.| `category`|Varies|
-|`taskSlot/idle/count`|Number of idle task slots per emission period. This
metric is only available if the `TaskSlotCountStatsMonitor` module is
included.| `category`|Varies|
-|`taskSlot/used/count`|Number of busy task slots per emission period. This
metric is only available if the `TaskSlotCountStatsMonitor` module is
included.| `category`|Varies|
-|`taskSlot/lazy/count`|Number of total task slots in lazy marked Middle
Managers and Indexers per emission period. This metric is only available if the
`TaskSlotCountStatsMonitor` module is included.| `category`|Varies|
-|`taskSlot/blacklisted/count`|Number of total task slots in blacklisted Middle
Managers and Indexers per emission period. This metric is only available if the
`TaskSlotCountStatsMonitor` module is included.| `category`|Varies|
-|`worker/task/failed/count`|Number of failed tasks run on the reporting worker
per emission period. This metric is only available if the
`WorkerTaskCountStatsMonitor` module is included, and is only supported for
Middle Manager nodes.| `category`, `workerVersion`|Varies|
-|`worker/task/success/count`|Number of successful tasks run on the reporting
worker per emission period. This metric is only available if the
`WorkerTaskCountStatsMonitor` module is included, and is only supported for
Middle Manager nodes.| `category`,`workerVersion`|Varies|
-|`worker/taskSlot/idle/count`|Number of idle task slots on the reporting
worker per emission period. This metric is only available if the
`WorkerTaskCountStatsMonitor` module is included, and is only supported for
Middle Manager nodes.| `category`, `workerVersion`|Varies|
-|`worker/taskSlot/total/count`|Number of total task slots on the reporting
worker per emission period. This metric is only available if the
`WorkerTaskCountStatsMonitor` module is included.| `category`,
`workerVersion`|Varies|
-|`worker/taskSlot/used/count`|Number of busy task slots on the reporting
worker per emission period. This metric is only available if the
`WorkerTaskCountStatsMonitor` module is included.| `category`,
`workerVersion`|Varies|
-|`worker/task/assigned/count`|Number of tasks assigned to an indexer per
emission period. This metric is only available if the
`WorkerTaskCountStatsMonitor` module is included.|`dataSource`|Varies|
-|`worker/task/completed/count`|Number of tasks completed by an indexer per
emission period. This metric is only available if the
`WorkerTaskCountStatsMonitor` module is included.|`dataSource`|Varies|
-|`worker/task/running/count`|Number of tasks running on an indexer per
emission period. This metric is only available if the
`WorkerTaskCountStatsMonitor` module is included.|`dataSource`|Varies|
+| Metric | Description
|Dimensions|Normal value|
+|------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------|------------|
+| `task/run/time` | Milliseconds taken to run a task.
| `dataSource`, `taskId`, `taskType`, `groupId`, `taskStatus`,
`tags`|Varies|
+| `task/pending/time` | Milliseconds taken for a task to
wait for running.
| `dataSource`, `taskId`, `taskType`, `groupId`,
`tags`|Varies|
+| `task/action/log/time` | Milliseconds taken to log a task
action to the audit log.
| `dataSource`, `taskId`, `taskType`, `groupId`,
`taskActionType`, `tags`|< 1000 (subsecond)|
+| `task/action/run/time` | Milliseconds taken to execute a
task action.
| `dataSource`, `taskId`, `taskType`, `groupId`,
`taskActionType`, `tags`|Varies from subsecond to a few seconds, based on
action type.|
+| `task/action/success/count` | Number of task actions that were
executed successfully during the emission period. Currently only being emitted
for [batched `segmentAllocate`
actions](../ingestion/tasks.md#batching-segmentallocate-actions). |
`dataSource`, `taskId`, `taskType`, `groupId`, `taskActionType`, `tags`|Varies|
+| `task/action/failed/count` | Number of task actions that
failed during the emission period. Currently only being emitted for [batched
`segmentAllocate`
actions](../ingestion/tasks.md#batching-segmentallocate-actions).
| `dataSource`, `taskId`, `taskType`, `groupId`, `taskActionType`,
`tags`|Varies|
+| `task/action/batch/queueTime` | Milliseconds spent by a batch of
task actions in queue. Currently only being emitted for [batched
`segmentAllocate`
actions](../ingestion/tasks.md#batching-segmentallocate-actions).
| `dataSource`, `taskActionType`, `interval`|Varies based on the
`batchAllocationWaitTime` and number of batches in queue.|
+| `task/action/batch/runTime` | Milliseconds taken to execute a
batch of task actions. Currently only being emitted for [batched
`segmentAllocate`
actions](../ingestion/tasks.md#batching-segmentallocate-actions).
| `dataSource`, `taskActionType`, `interval`|Varies from
subsecond to a few seconds, based on action type and batch size.|
+| `task/action/batch/size` | Number of task actions in a batch
that was executed during the emission period. Currently only being emitted for
[batched `segmentAllocate`
actions](../ingestion/tasks.md#batching-segmentallocate-actions). |
`dataSource`, `taskActionType`, `interval`|Varies based on number of concurrent
task actions.|
+| `task/action/batch/attempts` | Number of execution attempts for
a single batch of task actions. Currently only being emitted for [batched
`segmentAllocate`
actions](../ingestion/tasks.md#batching-segmentallocate-actions).
| `dataSource`, `taskActionType`, `interval`|1 if there are no failures
or retries.|
+| `task/segmentAvailability/wait/time` | The amount of milliseconds a
batch indexing task waited for newly created segments to become available for
querying.
| `dataSource`, `taskType`, `groupId`, `taskId`,
`segmentAvailabilityConfirmed`, `tags`|Varies|
+| `segment/added/bytes` | Size in bytes of new segments
created.
| `dataSource`, `taskId`, `taskType`, `groupId`,
`interval`, `tags`|Varies|
+| `segment/moved/bytes` | Size in bytes of segments
moved/archived via the Move Task.
| `dataSource`, `taskId`, `taskType`, `groupId`,
`interval`, `tags`|Varies|
+| `segment/nuked/bytes` | Size in bytes of segments deleted
via the Kill Task.
| `dataSource`, `taskId`, `taskType`, `groupId`, `interval`,
`tags`|Varies|
+| `task/success/count` | Number of successful tasks per
emission period. This metric is only available if the `TaskCountStatsMonitor`
module is included.
| `dataSource`|Varies|
+| `task/failed/count` | Number of failed tasks per
emission period. This metric is only available if the `TaskCountStatsMonitor`
module is included.
|`dataSource`|Varies|
+| `task/running/count` | Number of current running tasks.
This metric is only available if the `TaskCountStatsMonitor` module is
included.
|`dataSource`|Varies|
+| `task/pending/count` | Number of current pending tasks.
This metric is only available if the `TaskCountStatsMonitor` module is
included.
|`dataSource`|Varies|
+| `task/waiting/count` | Number of current waiting tasks.
This metric is only available if the `TaskCountStatsMonitor` module is
included.
|`dataSource`|Varies|
+| `taskSlot/total/count` | Number of total task slots per
emission period. This metric is only available if the
`TaskSlotCountStatsMonitor` module is included.
| `category`|Varies|
+| `taskSlot/idle/count` | Number of idle task slots per
emission period. This metric is only available if the
`TaskSlotCountStatsMonitor` module is included.
| `category`|Varies|
+| `taskSlot/used/count` | Number of busy task slots per
emission period. This metric is only available if the
`TaskSlotCountStatsMonitor` module is included.
| `category`|Varies|
+| `taskSlot/lazy/count` | Number of total task slots in
lazy marked Middle Managers and Indexers per emission period. This metric is
only available if the `TaskSlotCountStatsMonitor` module is included.
| `category`|Varies|
+| `taskSlot/blacklisted/count` | Number of total task slots in
blacklisted Middle Managers and Indexers per emission period. This metric is
only available if the `TaskSlotCountStatsMonitor` module is included.
| `category`|Varies|
+| `worker/task/failed/count` | Number of failed tasks run on the
reporting worker per emission period. This metric is only available if the
`WorkerTaskCountStatsMonitor` module is included, and is only supported for
Middle Manager nodes. | `category`, `workerVersion`|Varies|
+| `worker/task/success/count` | Number of successful tasks run on
the reporting worker per emission period. This metric is only available if the
`WorkerTaskCountStatsMonitor` module is included, and is only supported for
Middle Manager nodes. | `category`,`workerVersion`|Varies|
+| `worker/taskSlot/idle/count` | Number of idle task slots on the
reporting worker per emission period. This metric is only available if the
`WorkerTaskCountStatsMonitor` module is included, and is only supported for
Middle Manager nodes. | `category`, `workerVersion`|Varies|
+| `worker/taskSlot/total/count` | Number of total task slots on the
reporting worker per emission period. This metric is only available if the
`WorkerTaskCountStatsMonitor` module is included.
| `category`, `workerVersion`|Varies|
+| `worker/taskSlot/used/count` | Number of busy task slots on the
reporting worker per emission period. This metric is only available if the
`WorkerTaskCountStatsMonitor` module is included.
| `category`, `workerVersion`|Varies|
+| `worker/task/assigned/count` | Number of tasks assigned to an
indexer per emission period. This metric is only available if the
`WorkerTaskCountStatsMonitor` module is included.
|`dataSource`|Varies|
+| `worker/task/completed/count` | Number of tasks completed by an
indexer per emission period. This metric is only available if the
`WorkerTaskCountStatsMonitor` module is included.
|`dataSource`|Varies|
+| `worker/task/running/count` | Number of tasks running on an
indexer per emission period. This metric is only available if the
`WorkerTaskCountStatsMonitor` module is included.
|`dataSource`|Varies|
+| `s3upload/threadPool/taskQueuedDuration` | Milliseconds spent by a task in
queue before it starts uploading chunk to S3 when durable storage is enabled.
||Varies|
+| `s3upload/threadPool/queuedTasks` | The number of tasks that are
currently queued and waiting to upload chunks to S3 when durable storage is
enabled.
||Varies|
Review Comment:
> @kfaraz Karan suggested (in my previous PR) that it would be too noisy to
log it, so I removed it.
There is already an info log line in `RetryableS3OutputStream.close()`, line
213. You can just include the upload ID and path in that message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
