This is an automated email from the ASF dual-hosted git repository.
ethanfeng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new 402d23d0e [CELEBORN-1223] Align master and worker metrics of document
with MasterSource and WorkerSource
402d23d0e is described below
commit 402d23d0ea6d8bdcad412322610b6b7d41013812
Author: SteNicholas <[email protected]>
AuthorDate: Tue Jan 16 16:19:39 2024 +0800
[CELEBORN-1223] Align master and worker metrics of document with
MasterSource and WorkerSource
### What changes were proposed in this pull request?
Align master and worker metrics of document with `MasterSource` and
`WorkerSource` in `METRICS.md` and `monitoring.md`.
### Why are the changes needed?
Metrics of master and worker is inconsistent with `MasterSource` and
`WorkerSource` at present. It is recommended to align master and worker metrics
of document with `MasterSource` and `WorkerSource`:
- PushDataHandshakeFailCount
- RegionStartFailCount
- RegionFinishFailCount
- PrimaryPushDataHandshakeTime
- ReplicaPushDataHandshakeTime
- PrimaryRegionStartTime
- ReplicaRegionStartTime
- PrimaryRegionFinishTime
- ReplicaRegionFinishTime
- ActiveConnectionCount
- BufferStreamReadBuffer
- ReadBufferDispatcherRequestsLength
- ReadBufferAllocatedCount
- CreditStreamCount
- ActiveMapPartitionCount
- DeviceOSFreeBytes
- DeviceOSTotalBytes
- DeviceCelebornFreeBytes
- DeviceCelebornTotalBytes
- PotentialConsumeSpeed
- UserProduceSpeed
- WorkerConsumeSpeed
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
No.
Closes #2226 from SteNicholas/CELEBORN-1223.
Authored-by: SteNicholas <[email protected]>
Signed-off-by: mingji <[email protected]>
---
METRICS.md | 196 ++++++++++++++++++++++++++++++-----------------------
docs/monitoring.md | 51 +++++++-------
2 files changed, 136 insertions(+), 111 deletions(-)
diff --git a/METRICS.md b/METRICS.md
index 781390ee9..db54580c5 100644
--- a/METRICS.md
+++ b/METRICS.md
@@ -74,92 +74,116 @@ Here is an example of Grafana dashboard importing.
## Details
-| MetricName | Scope |
Description
|
-|:--------------------------------------:|:-----------------:|:---------------------------------------------------------------------------------------------------------------:|
-| WorkerCount | master |
The count of active workers.
|
-| ExcludedWorkerCount | master |
The count of workers in excluded list.
|
-| OfferSlotsTime | master |
The time of offer slots.
|
-| PartitionSize | master | The
estimated partition size of last 20 flush window whose length is 15 seconds by
defaults. |
-| RegisteredShuffleCount | master and worker |
The value means count of registered shuffle.
|
-| RunningApplicationCount | master and worker |
The value means count of running applications.
|
-| ActiveShuffleSize | master and worker | The value
means the active shuffle size for workers or a worker including master replica
and slave replica. |
-| ActiveShuffleFileCount | master and worker | The value
means the active shuffle size for workers or a worker including master replica
and slave replica. |
-| diskFileCount | master and worker |
The count of disk files consumption by each user.
|
-| diskBytesWritten | master and worker |
The amount of disk files consumption by each user.
|
-| hdfsFileCount | master and worker |
The count of hdfs files consumption by each user.
|
-| hdfsBytesWritten | master and worker |
The amount of hdfs files consumption by each user.
|
-| CommitFilesTime | worker |
CommitFiles means flush and close a shuffle partition file.
|
-| ReserveSlotsTime | worker |
ReserveSlots means acquire a disk buffer and record partition location.
|
-| FlushDataTime | worker |
FlushData means flush a disk buffer to disk.
|
-| OpenStreamTime | worker |
OpenStream means read a shuffle file and send client about chunks size and
stream index. |
-| FetchChunkTime | worker |
FetchChunk means read a chunk from a shuffle file and send to client.
|
-| ChunkStreamCount | worker |
The stream count for reduce partition reading streams in current worker.
|
-| OpenStreamFailCount | worker |
The count of opening stream failed in current worker.
|
-| FetchChunkFailCount | worker |
The count of fetching chunk failed in current worker.
|
-| PrimaryPushDataTime | worker |
PrimaryPushData means handle pushdata of primary partition location.
|
-| ReplicaPushDataTime | worker |
ReplicaPushData means handle pushdata of replica partition location.
|
-| WriteDataHardSplitCount | worker | The
count of writing PushData or PushMergedData to HARD_SPLIT partition in current
worker. |
-| WriteDataFailCount | worker |
The count of writing PushData or PushMergedData failed in current worker.
|
-| ReplicateDataFailCount | worker |
The count of replicating PushData or PushMergedData failed in current worker.
|
-| ReplicateDataWriteFailCount | worker | The count
of replicating PushData or PushMergedData failed caused by write failure in
peer worker. |
-| ReplicateDataCreateConnectionFailCount | worker | The count of
replicating PushData or PushMergedData failed caused by creating connection
failed in peer worker. |
-| ReplicateDataConnectionExceptionCount | worker | The count of
replicating PushData or PushMergedData failed caused by connection exception in
peer worker. |
-| ReplicateDataTimeoutCount | worker | The
count of replicating PushData or PushMergedData failed caused by push timeout
in peer worker. |
-| TakeBufferTime | worker |
TakeBuffer means get a disk buffer from disk flusher.
|
-| SlotsAllocated | worker |
Slots allocated in last hour
|
-| NettyMemory | worker |
The value measures all kinds of transport memory used by netty.
|
-| SortTime | worker |
SortTime measures the time used by sorting a shuffle file.
|
-| SortMemory | worker |
SortMemory means total reserved memory for sorting shuffle files .
|
-| SortingFiles | worker |
This value means the count of sorting shuffle files.
|
-| SortedFiles | worker |
This value means the count of sorted shuffle files.
|
-| SortedFileSize | worker |
This value means the count of sorted shuffle files 's total size.
|
-| DiskBuffer | worker | Disk buffers
are part of netty used memory, means data need to write to disk but haven't
been written to disk. |
-| PausePushDataTime | worker |
PausePushData means stop receiving data from client.
|
-| PausePushDataAndReplicateTime | worker |
PausePushDataAndReplicate means stop receiving data from client and other
workers. |
-| PausePushData | worker |
The count of stopping receiving data from client in current worker.
|
-| PausePushDataAndReplicate | worker |
The count of stopping receiving data from client and other workers in current
worker. |
-| jvm_gc_count | JVM |
The GC count of each garbage collector.
|
-| jvm_gc_time | JVM |
The GC cost time of each garbage collector.
|
-| jvm_memory_heap_init | JVM |
The amount of heap init memory.
|
-| jvm_memory_heap_max | JVM |
The amount of heap max memory.
|
-| jvm_memory_heap_used | JVM |
The amount of heap used memory.
|
-| jvm_memory_heap_committed | JVM |
The amount of heap committed memory.
|
-| jvm_memory_heap_usage | JVM |
The percentage of heap memory usage.
|
-| jvm_memory_non_heap_init | JVM |
The amount of non-heap init memory.
|
-| jvm_memory_non_heap_max | JVM |
The amount of non-heap max memory.
|
-| jvm_memory_non_heap_used | JVM |
The amount of non-heap uesd memory.
|
-| jvm_memory_non_heap_committed | JVM |
The amount of non-heap committed memory.
|
-| jvm_memory_non_heap_usage | JVM |
The percentage of non-heap memory usage.
|
-| jvm_memory_pools_init | JVM |
The amount of each memory pool's init memory.
|
-| jvm_memory_pools_max | JVM |
The amount of each memory pool's max memory.
|
-| jvm_memory_pools_used | JVM |
The amount of each memory pool's used memory.
|
-| jvm_memory_pools_committed | JVM |
The amount of each memory pool's committed memory.
|
-| jvm_memory_pools_used_after_gc | JVM |
The amount of each memory pool's used memory after GC.
|
-| jvm_memory_pools_usage | JVM |
The percentage of each memory pool's memory usage.
|
-| jvm_memory_total_init | JVM |
The amount of total init memory.
|
-| jvm_memory_total_max | JVM |
The amount of total max memory.
|
-| jvm_memory_total_used | JVM |
The amount of total used memory.
|
-| jvm_memory_total_committed | JVM |
The amount of each memory pool's committed memory.
|
-| jvm_direct_capacity | JVM |
An estimate of the total capacity of the buffers in this pool
|
-| jvm_direct_count | JVM |
An estimate of the number of buffers in the pool
|
-| jvm_direct_used | JVM |
An estimate of the memory that JVM is using for this buffer pool
|
-| jvm_mapped_capacity | JVM |
An estimate of the total capacity of the buffers in this pool
|
-| jvm_mapped_count | JVM |
An estimate of the number of buffers in the pool
|
-| jvm_mapped_used | JVM |
An estimate of the memory that JVM is using for this buffer pool
|
-| jvm_thread_count | JVM |
The current number of threads.
|
-| jvm_thread_daemon_count | JVM |
The current number of daemon threads.
|
-| jvm_thread_blocked_count | JVM |
The current number of threads having blocked state.
|
-| jvm_thread_deadlock_count | JVM |
The current number of threads having deadlock state.
|
-| jvm_thread_new_count | JVM |
The current number of threads having new state.
|
-| jvm_thread_runnable_count | JVM |
The current number of threads having runnable state.
|
-| jvm_thread_terminated_count | JVM |
The current number of threads having terminated state.
|
-| jvm_thread_timed_waiting_count | JVM |
The current number of threads having timed_waiting state.
|
-| jvm_thread_waiting_count | JVM |
The current number of threads having waiting state.
|
-| jvm_classloader_loaded | JVM |
The total number of classes loaded since the start of the JVM.
|
-| jvm_classloader_unloaded | JVM |
The total number of classes unloaded since the start of the JVM.
|
-| JVMCPUTime | system |
The JVM costs cpu time.
|
-| AvailableProcessors | system |
The amount of system available processors.
|
-| LastMinuteSystemLoad | system |
The last minute load of system.
|
+| MetricName | Scope |
Description
|
+|:--------------------------------------:|:-----------------:|:-----------------------------------------------------------------------------------------------------------------:|
+| RegisteredShuffleCount | master and worker |
The value means count of registered shuffle.
|
+| RunningApplicationCount | master and worker |
The value means count of running applications.
|
+| ActiveShuffleSize | master and worker | The value
means the active shuffle size for workers or a worker including master replica
and slave replica. |
+| ActiveShuffleFileCount | master and worker | The value means
the active shuffle file count for workers or a worker including master replica
and slave replica. |
+| diskFileCount | master and worker |
The count of disk files consumption by each user.
|
+| diskBytesWritten | master and worker |
The amount of disk files consumption by each user.
|
+| hdfsFileCount | master and worker |
The count of hdfs files consumption by each user.
|
+| hdfsBytesWritten | master and worker |
The amount of hdfs files consumption by each user.
|
+| WorkerCount | master |
The count of active workers.
|
+| LostWorkers | master |
The count of workers in lost list.
|
+| ExcludedWorkerCount | master |
The count of workers in excluded list.
|
+| IsActiveMaster | master |
Whether the current master is active.
|
+| PartitionSize | master | The
estimated partition size of last 20 flush window whose length is 15 seconds by
defaults. |
+| OfferSlotsTime | master |
The time of offer slots.
|
+| OpenStreamTime | worker |
OpenStream means read a shuffle file and send client about chunks size and
stream index. |
+| FetchChunkTime | worker |
FetchChunk means read a chunk from a shuffle file and send to client.
|
+| ChunkStreamCount | worker |
The stream count for reduce partition reading streams in current worker.
|
+| OpenStreamFailCount | worker |
The count of opening stream failed in current worker.
|
+| FetchChunkFailCount | worker |
The count of fetching chunk failed in current worker.
|
+| PrimaryPushDataTime | worker |
PrimaryPushData means handle PushData of primary partition location.
|
+| ReplicaPushDataTime | worker |
ReplicaPushData means handle PushData of replica partition location.
|
+| WriteDataHardSplitCount | worker | The
count of writing PushData or PushMergedData to HARD_SPLIT partition in current
worker. |
+| WriteDataFailCount | worker |
The count of writing PushData or PushMergedData failed in current worker.
|
+| ReplicateDataFailCount | worker |
The count of replicating PushData or PushMergedData failed in current worker.
|
+| ReplicateDataWriteFailCount | worker | The
count of replicating PushData or PushMergedData failed caused by write failure
in peer worker. |
+| ReplicateDataCreateConnectionFailCount | worker | The count of
replicating PushData or PushMergedData failed caused by creating connection
failed in peer worker. |
+| ReplicateDataConnectionExceptionCount | worker | The count
of replicating PushData or PushMergedData failed caused by connection exception
in peer worker. |
+| ReplicateDataTimeoutCount | worker | The
count of replicating PushData or PushMergedData failed caused by push timeout
in peer worker. |
+| PushDataHandshakeFailCount | worker |
The count of PushDataHandshake failed in current worker.
|
+| RegionStartFailCount | worker |
The count of RegionStart failed in current worker.
|
+| RegionFinishFailCount | worker |
The count of RegionFinish failed in current worker.
|
+| PrimaryPushDataHandshakeTime | worker |
PrimaryPushDataHandshake means handle PushData of primary partition location.
|
+| ReplicaPushDataHandshakeTime | worker |
ReplicaPushDataHandshake means handle PushData of replica partition location.
|
+| PrimaryRegionStartTime | worker |
PrimaryRegionStart means handle RegionStart of primary partition location.
|
+| ReplicaRegionStartTime | worker |
ReplicaRegionStart means handle RegionStart of replica partition location.
|
+| PrimaryRegionFinishTime | worker |
PrimaryRegionFinish means handle RegionFinish of primary partition location.
|
+| ReplicaRegionFinishTime | worker |
ReplicaRegionFinish means handle RegionFinish of replica partition location.
|
+| PausePushDataTime | worker |
PausePushData means stop receiving data from client.
|
+| PausePushDataAndReplicateTime | worker |
PausePushDataAndReplicate means stop receiving data from client and other
workers. |
+| PausePushData | worker |
The count of stopping receiving data from client in current worker.
|
+| PausePushDataAndReplicate | worker |
The count of stopping receiving data from client and other workers in current
worker. |
+| TakeBufferTime | worker |
TakeBuffer means get a disk buffer from disk flusher.
|
+| FlushDataTime | worker |
FlushData means flush a disk buffer to disk.
|
+| CommitFilesTime | worker |
CommitFiles means flush and close a shuffle partition file.
|
+| SlotsAllocated | worker |
Slots allocated in last hour
|
+| ReserveSlotsTime | worker |
ReserveSlots means acquire a disk buffer and record partition location.
|
+| ActiveConnectionCount | worker |
The value means count of active network connection.
|
+| NettyMemory | worker |
The value measures all kinds of transport memory used by netty.
|
+| SortTime | worker |
SortTime measures the time used by sorting a shuffle file.
|
+| SortMemory | worker |
SortMemory means total reserved memory for sorting shuffle files .
|
+| SortingFiles | worker |
This value means the count of sorting shuffle files.
|
+| SortedFiles | worker |
This value means the count of sorted shuffle files.
|
+| SortedFileSize | worker |
This value means the count of sorted shuffle files 's total size.
|
+| DiskBuffer | worker | Disk buffers
are part of netty used memory, means data need to write to disk but haven't
been written to disk. |
+| BufferStreamReadBuffer | worker |
This value means memory used by credit stream read buffer.
|
+| ReadBufferDispatcherRequestsLength | worker |
This value means the queue size of read buffer allocation requests.
|
+| ReadBufferAllocatedCount | worker |
This value means count of allocated read buffer.
|
+| CreditStreamCount | worker |
This value means count of stream for map partition reading streams.
|
+| ActiveMapPartitionCount | worker |
This value means count of active map partition reading streams.
|
+| DeviceOSFreeBytes | worker |
This value means actual usable space of OS for device monitor.
|
+| DeviceOSTotalBytes | worker |
This value means total usable space of OS for device monitor.
|
+| DeviceCelebornFreeBytes | worker |
This value means actual usable space of Celeborn for device monitor.
|
+| DeviceCelebornTotalBytes | worker |
This value means configured usable space of Celeborn for device monitor.
|
+| PotentialConsumeSpeed | worker |
This value means speed of potential consumption for congestion control.
|
+| UserProduceSpeed | worker |
This value means speed of user production for congestion control.
|
+| WorkerConsumeSpeed | worker |
This value means speed of worker consumption for congestion control.
|
+| jvm_gc_count | JVM |
The GC count of each garbage collector.
|
+| jvm_gc_time | JVM |
The GC cost time of each garbage collector.
|
+| jvm_memory_heap_init | JVM |
The amount of heap init memory.
|
+| jvm_memory_heap_max | JVM |
The amount of heap max memory.
|
+| jvm_memory_heap_used | JVM |
The amount of heap used memory.
|
+| jvm_memory_heap_committed | JVM |
The amount of heap committed memory.
|
+| jvm_memory_heap_usage | JVM |
The percentage of heap memory usage.
|
+| jvm_memory_non_heap_init | JVM |
The amount of non-heap init memory.
|
+| jvm_memory_non_heap_max | JVM |
The amount of non-heap max memory.
|
+| jvm_memory_non_heap_used | JVM |
The amount of non-heap uesd memory.
|
+| jvm_memory_non_heap_committed | JVM |
The amount of non-heap committed memory.
|
+| jvm_memory_non_heap_usage | JVM |
The percentage of non-heap memory usage.
|
+| jvm_memory_pools_init | JVM |
The amount of each memory pool's init memory.
|
+| jvm_memory_pools_max | JVM |
The amount of each memory pool's max memory.
|
+| jvm_memory_pools_used | JVM |
The amount of each memory pool's used memory.
|
+| jvm_memory_pools_committed | JVM |
The amount of each memory pool's committed memory.
|
+| jvm_memory_pools_used_after_gc | JVM |
The amount of each memory pool's used memory after GC.
|
+| jvm_memory_pools_usage | JVM |
The percentage of each memory pool's memory usage.
|
+| jvm_memory_total_init | JVM |
The amount of total init memory.
|
+| jvm_memory_total_max | JVM |
The amount of total max memory.
|
+| jvm_memory_total_used | JVM |
The amount of total used memory.
|
+| jvm_memory_total_committed | JVM |
The amount of each memory pool's committed memory.
|
+| jvm_direct_capacity | JVM |
An estimate of the total capacity of the buffers in this pool
|
+| jvm_direct_count | JVM |
An estimate of the number of buffers in the pool
|
+| jvm_direct_used | JVM |
An estimate of the memory that JVM is using for this buffer pool
|
+| jvm_mapped_capacity | JVM |
An estimate of the total capacity of the buffers in this pool
|
+| jvm_mapped_count | JVM |
An estimate of the number of buffers in the pool
|
+| jvm_mapped_used | JVM |
An estimate of the memory that JVM is using for this buffer pool
|
+| jvm_thread_count | JVM |
The current number of threads.
|
+| jvm_thread_daemon_count | JVM |
The current number of daemon threads.
|
+| jvm_thread_blocked_count | JVM |
The current number of threads having blocked state.
|
+| jvm_thread_deadlock_count | JVM |
The current number of threads having deadlock state.
|
+| jvm_thread_new_count | JVM |
The current number of threads having new state.
|
+| jvm_thread_runnable_count | JVM |
The current number of threads having runnable state.
|
+| jvm_thread_terminated_count | JVM |
The current number of threads having terminated state.
|
+| jvm_thread_timed_waiting_count | JVM |
The current number of threads having timed_waiting state.
|
+| jvm_thread_waiting_count | JVM |
The current number of threads having waiting state.
|
+| jvm_classloader_loaded | JVM |
The total number of classes loaded since the start of the JVM.
|
+| jvm_classloader_unloaded | JVM |
The total number of classes unloaded since the start of the JVM.
|
+| JVMCPUTime | system |
The JVM costs cpu time.
|
+| AvailableProcessors | system |
The amount of system available processors.
|
+| LastMinuteSystemLoad | system |
The last minute load of system.
|
## Implementation
diff --git a/docs/monitoring.md b/docs/monitoring.md
index b172f496e..73515f150 100644
--- a/docs/monitoring.md
+++ b/docs/monitoring.md
@@ -91,18 +91,18 @@ reported in the list.
These metrics are exposed by Celeborn master.
- namespace=master
- - WorkerCount
- - LostWorkers
- - ExcludedWorkerCount
- RegisteredShuffleCount
- RunningApplicationCount
- - IsActiveMaster
- - PartitionSize
- - The size of estimated shuffle partition.
- ActiveShuffleSize
- The active shuffle size of workers.
- ActiveShuffleFileCount
- The active shuffle file count of workers.
+ - WorkerCount
+ - LostWorkers
+ - ExcludedWorkerCount
+ - IsActiveMaster
+ - PartitionSize
+ - The size of estimated shuffle partition.
- OfferSlotsTime
- The time for masters to handle `RequestSlots` request when
registering shuffle.
@@ -131,11 +131,12 @@ These metrics are exposed by Celeborn master.
These metrics are exposed by Celeborn worker.
- namespace=worker
- - CommitFilesTime
- - The time for a worker to flush buffers and close files related to
specified shuffle.
- - ReserveSlotsTime
- - FlushDataTime
- - The time for a worker to write a buffer which is 256KB by default to
storage.
+ - RegisteredShuffleCount
+ - RunningApplicationCount
+ - ActiveShuffleSize
+ - The active shuffle size of a worker including master replica and
slave replica.
+ - ActiveShuffleFileCount
+ - The active shuffle file count of a worker including master replica
and slave replica.
- OpenStreamTime
- The time for a worker to process openStream RPC and return
StreamHandle.
- FetchChunkTime
@@ -164,11 +165,23 @@ These metrics are exposed by Celeborn worker.
- ReplicaRegionStartTime
- PrimaryRegionFinishTime
- ReplicaRegionFinishTime
+ - PausePushDataTime
+ - The time for a worker to stop receiving pushData from clients
because of back pressure.
+ - PausePushDataAndReplicateTime
+ - The time for a worker to stop receiving pushData from clients and
other workers because of back pressure.
+ - PausePushData
+ - The count for a worker to stop receiving pushData from clients
because of back pressure.
+ - PausePushDataAndReplicate
+ - The count for a worker to stop receiving pushData from clients and
other workers because of back pressure.
- TakeBufferTime
- The time for a worker to take out a buffer from a disk flusher.
- - RegisteredShuffleCount
- - RunningApplicationCount
+ - FlushDataTime
+ - The time for a worker to write a buffer which is 256KB by default to
storage.
+ - CommitFilesTime
+ - The time for a worker to flush buffers and close files related to
specified shuffle.
- SlotsAllocated
+ - ReserveSlotsTime
+ - ActiveConnectionCount
- NettyMemory
- The total amount of off-heap memory used by celeborn worker.
- SortTime
@@ -180,14 +193,6 @@ These metrics are exposed by Celeborn worker.
- SortedFileSize
- DiskBuffer
- The memory occupied by pushData and pushMergedData which should be
written to disk.
- - PausePushDataTime
- - The time for a worker to stop receiving pushData from clients
because of back pressure.
- - PausePushDataAndReplicateTime
- - The time for a worker to stop receiving pushData from clients and
other workers because of back pressure.
- - PausePushData
- - The count for a worker to stop receiving pushData from clients
because of back pressure.
- - PausePushDataAndReplicate
- - The count for a worker to stop receiving pushData from clients and
other workers because of back pressure.
- BufferStreamReadBuffer
- The memory used by credit stream read buffer.
- ReadBufferDispatcherRequestsLength
@@ -204,10 +209,6 @@ These metrics are exposed by Celeborn worker.
- PotentialConsumeSpeed
- UserProduceSpeed
- WorkerConsumeSpeed
- - ActiveShuffleSize
- - The active shuffle size of a worker including master replica and
slave replica.
- - ActiveShuffleFileCount
- - The active shuffle file count of a worker including master replica
and slave replica.
- push_server_usedHeapMemory
- push_server_usedDirectMemory
- push_server_numAllocations
