This is an automated email from the ASF dual-hosted git repository.
benjobs pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/incubator-streampark.git
The following commit(s) were added to refs/heads/dev by this push:
new 3517c06f0 [Feature] Add dead link check (#1789)
3517c06f0 is described below
commit 3517c06f02540ab66a603ddba9a48ddc98ad9809
Author: ChunFu Wu <[email protected]>
AuthorDate: Mon Oct 10 09:30:59 2022 +0800
[Feature] Add dead link check (#1789)
* [Feature] Add dead link check
---
.dlc.json | 32 ++
.github/PULL_REQUEST_TEMPLATE.md | 2 +-
.github/workflows/maven.yml | 13 +-
.../streampark-jvm-profiler/metricDetails.md | 358 ++++++++++-----------
4 files changed, 214 insertions(+), 191 deletions(-)
diff --git a/.dlc.json b/.dlc.json
new file mode 100644
index 000000000..e80f2c3f0
--- /dev/null
+++ b/.dlc.json
@@ -0,0 +1,32 @@
+{
+ "ignorePatterns": [
+ {
+ "pattern": "^http://localhost";
+ },
+ {
+ "pattern": "^https://mvnrepository.com";
+ },
+ {
+ "pattern": "^https://img.shields.io";
+ },
+ {
+ "pattern": "^https://tokei.rs";
+ },
+ {
+ "pattern": "^https://json.org/";
+ },
+ {
+ "pattern": "^https://opencollective.com";
+ }
+ ],
+ "timeout": "30s",
+ "retryOn429": true,
+ "retryCount": 10,
+ "fallbackRetryDelay": "1000s",
+ "aliveStatusCodes": [
+ 0,
+ 200,
+ 401,
+ 403
+ ]
+}
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 9c50837d4..267d75368 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -14,7 +14,7 @@ Problem Summary:
## What is changed and how it works?
-Proposal: [xxx](url) <!-- REMOVE this line if not applicable -->
+Proposal: url address <!-- REMOVE this line if not applicable -->
Feel free to ping committers for the review!
diff --git a/.github/workflows/maven.yml b/.github/workflows/maven.yml
index b3dcc7981..834f4f938 100644
--- a/.github/workflows/maven.yml
+++ b/.github/workflows/maven.yml
@@ -45,7 +45,18 @@ jobs:
submodules: true
- name: Check license header
uses: apache/skywalking-eyes/header@main
-
+ dead-link:
+ if: github.repository == 'apache/incubator-streampark'
+ name: Dead links
+ runs-on: ubuntu-latest
+ timeout-minutes: 20
+ steps:
+ - uses: actions/checkout@v3
+ - run: sudo npm install -g [email protected]
+ - run: |
+ for file in $(find . -name "*.md"); do
+ markdown-link-check -c .dlc.json -q "$file"
+ done
build:
runs-on: ubuntu-latest
name: Build with Scala_2.11
diff --git a/streampark-plugin/streampark-jvm-profiler/metricDetails.md
b/streampark-plugin/streampark-jvm-profiler/metricDetails.md
index 4ef785240..2e9d238ff 100644
--- a/streampark-plugin/streampark-jvm-profiler/metricDetails.md
+++ b/streampark-plugin/streampark-jvm-profiler/metricDetails.md
@@ -1,189 +1,169 @@
-# Details of Metrics Collected by the JVM-Profiler <a name="Title"></a>
-
-1. [CPU and Memory](#CPUAndMemory)
-2. [IO](#IO)
-3. [Process Info](#ProcessInfo)
-4. [Stack Trace](#StackTrace)
-
-## 1. CPU and Memory <a name="CPUAndMemory"></a>
-
-|**Name** |**
-Description** |**Sample
Value** |**
-Note**
|
-|--------------------------------------------------------|-----------------------------------------------------------------|----------------------------------------------|---------------------------------------------------------------------------------------|
-|time |Time series value
for grafana database. |1582504748538000000
|
|
-|appId |Application id of
the job that is running.
|application_1576797426048_311346 |If using yarn, this is the yarn
application id. |
-|bufferPools-direct-count |Total number of
direct byte buffers used at the recorded time. |55
|Depending on your application, this can be a smaller number
|
-|bufferPools-direct-memoryUsed |Total memory of the
direct byte buffers used in bytes. |18950
|An estimate. This value should be very similar to the total
capacity for a healthy JVM.|
-|bufferPools-direct-name |Name of the buffer
pool. |direct
|Direct buffer pools are always named direct.
|
-|bufferPools-direct-totalCapacity |Available capacity
in bytes in the direct buffer pool. |18949
|Capacity should be similar to memoryUsed for a healthy JVM.
|
-|bufferPools-mapped-count |Total number of
mapped byte buffers used at the recorded time. |0
|In general, spark applications will not require mapped buffer
pools. |
-|bufferPools-mapped-memoryUsed |Total memory of the
mapped byte buffers used in bytes. |0
|An estimate. This value should be very similar to the total
capacity for a healthy JVM.|
-|bufferPools-mapped-name |Name of the buffer
pool. |mapped
|Mapped buffer pools are always named mapped.
|
-|bufferPools-mapped-totalCapacity |Available capacity
in bytes in the mapped buffer pool. |0
|Capacity should be similar to memoryUsed for a healthy JVM.
|
-|epochMillis |System time in
milliseconds when this value was recorded. |1582504781508
|
|
-|gc-PSMarkSweep-collectionCount |Total number of
collections that have occurred for PS MarkSweep. |7
|-1 if collection count is undefined for this collector.
|
-|gc-PSMarkSweep-collectionTime |approximate
accumulated collection elapsed time in milliseconds. |914
|-1 if collection count is undefined for this collector.
|
-|gc-PSMarkSweep-name |Name of the
collector. |PS MarkSweep
|Constant value.
|
-|gc-PSScavenge-collectionCount |Total number of
collections that have occurred for PS Scavenge. |25
|-1 if collection count is undefined for this collector.
|
-|gc-PSScavenge-collectionTime |approximate
accumulated collection elapsed time in milliseconds. |1206
|-1 if collection count is undefined for this collector.
|
-|gc-PSScavenge-name |Name of the
collector. |PS Scavenge
|Constant value.
|
-|heapMemoryCommitted |Amount of memory in
bytes guaranteed for the JVM to use. |375914496
|
|
-|heapMemoryMax |Max amount of memory
in bytes that can be used |11453595648
|Also refer to spark.memory.fraction for the fraction managed by
spark |
-|heapMemoryTotalUsed |Total heap memory
used in bytes |86386032
|Total memory used should be less than committed and max heap
memory. |
-|host |Hostname of the
executor |bigdata-datanode1.com
|
|
-|memoryPools-CodeCache-name |Name of the code
cache. |Code Cache
|Constant value.
|
-|memoryPools-CodeCache-peakUsageCommitted |Peak committed
memory of this pool since the JVM was started |2555904
|
|
-|memoryPools-CodeCache-peakUsageMax |Peak value of
usageMax of this pool since the JVM was started |251658240
|
|
-|memoryPools-CodeCache-peakUsageUsed |Peak value of memory
of this pool used since the JVM was started.|582656
|
|
-|memoryPools-CodeCache-type |The type of this
memory pool. |Non-heap memory
|Code cache is a non-heap memory. So this value will always be
constant. |
-|memoryPools-CodeCache-usageCommitted |Total memory in
bytes committed for this pool. |2555904
|
|
-|memoryPools-CodeCache-usageMax |The maximum amount
of memory that can be used by this pool. |251658240
|-1 if undefined. Not guaranteed to be available if greater than
the committed memory. |
-|memoryPools-CodeCache-usageUsed |Amount of memory
used by this pool. |582656
|Having some code cache memory free ensures it is usable for JIT
compilation. |
-|memoryPools-CompressedClassSpace-name |Name of the
compressed class space. |Compressed Class Space
|Is constant. This space is allocated in native memory for
class metadata. |
-|memoryPools-CompressedClassSpace-peakUsageCommitted |Peak committed
memory of this pool since JVM was started. |1572864
|
|
-|memoryPools-CompressedClassSpace-peakUsageMax |Peak value of
usageMax of this pool since the JVM was started. |1073741824
|
|
-|memoryPools-CompressedClassSpace-peakUsageUsed |Peak value of usage
of this pool since the JVM was started. |1389800
|
|
-|memoryPools-CompressedClassSpace-type |Type of memory of
this memory pool. |Non-heap memory
|Compressed class space is native memory but is managed by the
JVM |
-|memoryPools-CompressedClassSpace-usageCommitted |Total memory in
bytes committed for this pool. |1572864
|
|
-|memoryPools-CompressedClassSpace-usageMax |The maximum amount
of memory that can be used by this pool |1073741824
|-1 if undefined. Not guaranteed to be available if greater than
the committed memory. |
-|memoryPools-CompressedClassSpace-usageUsed |Amount of memory
used by this pool. |1389800
|
|
-|memoryPools-Metaspace-name |Name of the
Metaspace. |Metaspace
|Is constant. Metaspace is always called metaspace.
|
-|memoryPools-Metaspace-peakUsageCommitted |Peak committed
memory of this pool since JVM was started. |12320768
|
|
-|memoryPools-Metaspace-peakUsageMax |Peak value of
usageMax of this pool since the JVM was started. |-1
|-1 if undefined.
|
-|memoryPools-Metaspace-peakUsageUsed |Peak value of usage
of this pool since the JVM was started. |11768752
|
|
-|memoryPools-Metaspace-type |Type of memory of
this memory pool. |Non-heap memory
|Metaspace is native memory (since JDK 8) but is managed by the
JVM |
-|memoryPools-Metaspace-usageCommitted |Total memory in
bytes committed for this pool. |12320768
|
|
-|memoryPools-Metaspace-usageMax |The maximum amount
of memory that can be used by this pool |-1
|-1 if undefined. Not guaranteed to be available if greater than
the committed memory. |
-|memoryPools-Metaspace-usageUsed |Amount of memory
used by this pool. |11768752
|The amount of space used by loaded classes.
|
-|memoryPools-PSEdenSpace-name |Name of the young
generation heap space. |PS Eden Space
|Is constant. PS Eden space is always called PS Eden Space.
|
-|memoryPools-PSEdenSpace-peakUsageCommitted |Peak committed
memory of this pool since JVM was started. |98041856
|
|
-|memoryPools-PSEdenSpace-peakUsageMax |Peak value of
usageMax of this pool since the JVM was started. |4262461440
|-1 if undefined.
|
-|memoryPools-PSEdenSpace-peakUsageUsed |Peak value of usage
of this pool since the JVM was started. |92270136
|
|
-|memoryPools-PSEdenSpace-type |Type of memory of
this memory pool. |Heap memory
|PS Eden space is a portion of the heap memory that is managed by
the JVM |
-|memoryPools-PSEdenSpace-usageCommitted |Total memory in
bytes committed for this pool. |98041856
|
|
-|memoryPools-PSEdenSpace-usageMax |The maximum amount
of memory that can be used by this pool |8589934592
|-1 if undefined. Not guaranteed to be available if greater than
the committed memory. |
-|memoryPools-PSEdenSpace-usageUsed |Amount of memory
used by this pool (and by newly loaded classes).|92270136
|A continuously increasing value indicates an unhealthy garbage
collection. |
-|memoryPools-PSOldGen-name |Name of the old
generation heap space. |PS Old Gen
|Is constant. PS Old Gen space is always called PS Old Gen.
|
-|memoryPools-PSOldGen-peakUsageCommitted |Peak committed
memory of this pool since JVM was started. |261619712
|
|
-|memoryPools-PSOldGen-peakUsageMax |Peak value of
usageMax of this pool since the JVM was started. |8589934592
|-1 if undefined.
|
-|memoryPools-PSOldGen-peakUsageUsed |Peak value of usage
of this pool since the JVM was started. |0
|
|
-|memoryPools-PSOldGen-type |Type of memory of
this memory pool. |Heap memory
|PS Old Gen space is a portion of the heap memory that is managed
by the JVM |
-|memoryPools-PSOldGen-usageCommitted |Total memory in
bytes committed for this pool. |261619712
|
|
-|memoryPools-PSOldGen-usageMax |The maximum amount
of memory that can be used by this pool |8589934592
|-1 if undefined. Not guaranteed to be available if greater than
the committed memory. |
-|memoryPools-PSOldGen-usageUsed |Amount of memory
used by this pool (by objects beyond threshold).|0
|A continuously increasing value indicates an unhealthy garbage
collection. |
-|memoryPools-PSSurvivorSpace-name |Name of the survivor
heap space. |PS Survivor Space
|Is constant. PS Survivor Space is always called PS Survivor Space.
|
-|memoryPools-PSSurvivorSpace-peakUsageCommitted |Peak committed
memory of this pool since JVM was started. |16252928
|
|
-|memoryPools-PSSurvivorSpace-peakUsageMax |Peak value of
usageMax of this pool since the JVM was started. |16252928
|-1 if undefined.
|
-|memoryPools-PSSurvivorSpace-peakUsageUsed |Peak value of usage
of this pool since the JVM was started. |0
|
|
-|memoryPools-PSSurvivorSpace-type |Type of memory of
this memory pool. |Heap memory
|PS Survivor Space is a portion of the heap memory that is
managed by the JVM |
-|memoryPools-PSSurvivorSpace-usageCommitted |Total memory in
bytes committed for this pool. |16252928
|
|
-|memoryPools-PSSurvivorSpace-usageMax |The maximum amount
of memory that can be used by this pool |16252928
|-1 if undefined. Not guaranteed to be available if greater than
the committed memory. |
-|memoryPools-PSSurvivorSpace-usageUsed |Amount of memory
used by this pool (and by newly loaded classes).|0
|A continuously increasing value indicates an unhealthy garbage
collection. |
-|name |Process name
(number)
|[email protected] |Also contains host name.
|
-|nonHeapMemoryCommitted |Amount of non heap
memory in bytes committed for the JVM to use. |16187392
|Total of committed of all non-heap memory pools.
|
-|nonHeapMemoryMax |Maximum amount of
memory in bytes that can be used. |-1
|-1 if undefined.
|
-|nonHeapMemoryTotalUsed |Amount of non heap
memory (metaspace, code cache) used. |13510368
|Total of usage of all non-heap memory pools.
|
-|processCpuLoad |The recent cpu usage
for the JVM process. |4.335671366438364e-9
|
|
-|processCpuTime |CPU time used by JVM
process in nanoseconds. |2340000000
|
|
-|processUuid |UUID of the JVM
process.
|48abf05b-dfd4-4069-86c6-8cdbc195bd40 |
|
-|processUuid_1 |UUID of the JVM
process.
|48abf05b-dfd4-4069-86c6-8cdbc195bd40 |Appears to be a duplicate
metric. |
-|role |Role of this spark
executor. |executor
|Can be either executor or driver.
|
-|systemCpuLoad |The recent cpu usage
for the whole system. |0.03703887412577616
|
|
-|vmHWM |Peak resident set
size ("High Water Mark") in bytes for the JVM. |142876672
|An indication of actual physical memory (RAM) usage in linux by
the JVM. |
-|vmPeak |Peak of the virtual
memory usage in bytes for the JVM. |15130554368
|
|
-|vmRSS |JVM physical memory
usage (resident set size) in bytes. |142876672
|
|
-|vmSize |Virtual memory size
for the JVM. |15130550272
|This memory includes swap space, so a large value can imply high
swap memory usage. |
-
-[Back to Top](#Title)
-
-## 2. IO <a name="IO"></a>
-
-|**Name** |**
-Description** |**Sample
Value** |**
-Notes**
|
-|--------------------------------------------------------|-----------------------------------------------------------------|----------------------------------------------|---------------------------------------------------------------------------------------|
-|time |Time series value
for grafana database. |1582504713009000000
|
|
-|appId |Application id for
the job that is running. |application_1576797426048_311346
|If using yarn, this is the yarn application id.
|
-|epochMillis |System time in
milliseconds when this value was recorded. |1582504713001
|
|
-|host |Hostname of the
executor. |bigdata-datanode1.com
|
|
-|name |Process name
(number).
|[email protected] |Includes the hostname as well.
|
-|processUuid |UUID of the JVM
process.
|89a256c7-96c3-4e48-b5ba-f55e886ba743 |
|
-|processUuid_1 |UUID of the JVM
process.
|89a256c7-96c3-4e48-b5ba-f55e886ba743 |Appears to be a duplicate
metric. |
-|role |Role of this spark
executor. |executor
|Can be either executor or driver.
|
-|self-io-rchar-1 |The number of
characters read by this JVM process. |8192340
|The "-1" is coming from the formatter for influx which was
used when writting this. |
-|self-io-read_bytes-1 |Attempt to count the
bytes read by this JVM process from storage.|1892352
|
|
-|self-io-wchar-1 |The number of
characters written to disk by this JVM process. |51016
|
|
-|self-io-write_bytes-1 |Attempt to count the
bytes written by JVM process to storage. |86016
|
|
-|stat-cpu-N |CPU entry found in
/proc/<pid>/stat. |cpu, cpu0, cpu1, cpu2 ...
|N is the CPU number starting at cpu-1 (i.e. cpu) which aggregates
all other CPU lines. |
-|stat-idle-N |Time the N-th CPU
spent in the idle task. |53679018
|Time is in USER_HZ (1/100th of a second).
|
-|stat-iowait-N |Time spent by N-th
CPU waiting for I/O to complete. |137645010
|
|
-|stat-nice-N |Time spent by N-th
CPU in user mode with low priority (nice). |104
|Low idle time with high nice time are indications of busy
background tasks running. |
-|stat-system-N |Time spent by N-th
CPU in system mode. |32700364
|Time spent at kernel level.
|
-|stat-user-N |Time spent by N-th
CPU at the user (application) level |279816973
|
|
-
-[Back to Top](#Title)
-
-## 3. Process Info <a name="ProcessInfo"></a>
-
-|**Name** |**
-Description** |**Sample
Value** |**
-Notes**
|
-|--------------------------------------------------------|-----------------------------------------------------------------|----------------------------------------------|---------------------------------------------------------------------------------------|
-|time |Time series value
for grafana database. |1582504713009000000
|
|
-|agentVersion |A constant value
defined in code. |1.0.0
|Agent version will always be 1.0.0.
|
-|appId |Application id for
the job that is running. |application_1576797426048_311346
|If using yarn, this is the yarn application id.
|
-|cmdline |Contains JVM command
executed and input arguments and classpath. |/usr/jdk64/jdk1.8.0_112/bin/java
-javaagent...|
|
-|epochMillis |System time in
milliseconds when this value was recorded. |1582504713001
|
|
-|host |Hostname of the
executor. |bigdata-datanode1.com
|
|
-|jvmClassPath |Contains JVM
classpath entries. |--user-class-path
.../jvm-profiler-1.0.0.jar |This entry is emtpy if cmdLine is not empty. This
is to avoid duplicate information. |
-|jvmInputArguments |Contains JVM input
arguments.
|metricInterval=5000,sampleInterval=5000 ... |This entry is emtpy if cmdLine
is not empty. This is to avoid duplicate information. |
-|name |Process name
(number).
|[email protected] |Includes the hostname as well.
|
-|processUuid |UUID of the JVM
process.
|89a256c7-96c3-4e48-b5ba-f55e886ba743 |
|
-|processUuid_1 |UUID of the JVM
process.
|89a256c7-96c3-4e48-b5ba-f55e886ba743 |Appears to be a duplicate
metric. |
-|role |Role of this spark
executor. |executor
|Can be either executor or driver.
|
-|xmxBytes |Maximum JVM heap
size in bytes. |12884901888
|
|
-
-[Back to Top](#Title)
-
-## 4. Stack Trace <a name="StackTrace"></a>
-
-|**Name** |**
-Description** |**Sample
Value** |**
-Notes**
|
-|--------------------------------------------------------|-----------------------------------------------------------------|----------------------------------------------|---------------------------------------------------------------------------------------|
-|time |Time series value
for grafana database. |1582504713009000000
|
|
-|appId |Application id for
the job that is running. |application_1576797426048_311346
|If using yarn, this is the yarn application id.
|
-|count |Number of elements
appended to the metrics buffer. |1
|This value is usually 1, but not guaranteed to be constant. It is
thread-safe. |
-|endEpoch |System time in
milliseconds after resetting the metrics buffer. |1582504712723
|The metrics buffer is an internal buffer used by this
profiling application. |
-|host |Hostname of the
executor. |bigdata-datanode1.com
|
|
-|name |Process name
(number).
|[email protected] |Includes the hostname as well.
|
-|processUuid |UUID of the JVM
process.
|89a256c7-96c3-4e48-b5ba-f55e886ba743 |
|
-|processUuid_1 |UUID of the JVM
process.
|89a256c7-96c3-4e48-b5ba-f55e886ba743 |Appears to be a duplicate
metric. |
-|role |Role of this spark
executor. |executor
|Can be either executor or driver.
|
-|stacktrace |Stacktrace starting
from bottom of the stacktrace.
|java.lang.Object.wait,java.lang.ref ... |Trims the top method (most
nested method) if the size is too large (800k characters) |
-|startEpoch |System time in
milliseconds before resetting the metrics buffer. |1582504710523
|The metrics buffer is an internal buffer used by this
profiling application. |
-|threadName |Name of the thread
for which stacktrace is displayed. |Signal Dispatcher
|
|
-|threadState |State of the thread
when this stacktrace was captured. |NEW, RUNNABLE, BLOCKED, WAITING
... |
|
-
-[Back to Top](#Title)
-
-## 5. Thread Info <a name="ThreadInfo"></a>
-
-|**Name** |**
-Description** |**Sample
Value** |**
-Notes**
|
-|--------------------------------------------------------|-----------------------------------------------------------------|----------------------------------------------|---------------------------------------------------------------------------------------|
-|time |Time series value
for grafana database. |1582504713009000000
|
|
-|epochMillis |System time in
milliseconds when this value was recorded. |1582504781508
|
|
-|host |Hostname of the
executor. |bigdata-datanode1.com
|
|
-|name |Process name
(number).
|[email protected] |Includes the hostname as well.
|
-|appId |Application id for
the job that is running. |application_1576797426048_311346
|If using yarn, this is the yarn application id.
|
-|processUuid |UUID of the JVM
process.
|89a256c7-96c3-4e48-b5ba-f55e886ba743 |
|
-|role |Role of this spark
executor. |executor
|Can be either executor or driver.
|
-|totalStartedThreadCount |Total Number
of Thread created so far since JVm Launch. |10
|
|
-|liveThreadCount |Number of thread
which are currently active. |8
|
|
-|newThreadCount |Number of thread
which are newly Created. |3
| Derived from last Cycle Total Threads.
|
-|peakThreadCount |the peak live thread
count since the JVM start |8
|
|
-
-[Back to Top](#Title)
+# Details of Metrics Collected by the JVM-Profiler
+
+1. CPU and Memory
+2. IO
+3. Process Info
+4. Stack Trace
+
+## 1. CPU and Memory
+
+|**Name** |**Description**
| **Sample Value**
|**Note**
|
+|-----------------------------------------------------|-----------------------------------------------------------------|---------------------------------------|---------------------------------------------------------------------------------------|
+|time |Time series value for
grafana database. | 1582504748538000000
|
|
+|appId |Application id of the
job that is running. | application_1576797426048_311346
|If using yarn, this is the yarn application id.
|
+|bufferPools-direct-count |Total number of direct
byte buffers used at the recorded time. | 55
|Depending on your application, this can be a smaller number
|
+|bufferPools-direct-memoryUsed |Total memory of the
direct byte buffers used in bytes. | 18950
|An estimate. This value should be very similar to the total capacity for
a healthy JVM.|
+|bufferPools-direct-name |Name of the buffer
pool. | direct
|Direct buffer pools are always named direct.
|
+|bufferPools-direct-totalCapacity |Available capacity in
bytes in the direct buffer pool. | 18949
|Capacity should be similar to memoryUsed for a healthy JVM.
|
+|bufferPools-mapped-count |Total number of mapped
byte buffers used at the recorded time. | 0
|In general, spark applications will not require mapped buffer pools.
|
+|bufferPools-mapped-memoryUsed |Total memory of the
mapped byte buffers used in bytes. | 0
|An estimate. This value should be very similar to the total capacity for
a healthy JVM.|
+|bufferPools-mapped-name |Name of the buffer
pool. | mapped
|Mapped buffer pools are always named mapped.
|
+|bufferPools-mapped-totalCapacity |Available capacity in
bytes in the mapped buffer pool. | 0
|Capacity should be similar to memoryUsed for a healthy JVM.
|
+|epochMillis |System time in
milliseconds when this value was recorded. | 1582504781508
|
|
+|gc-PSMarkSweep-collectionCount |Total number of
collections that have occurred for PS MarkSweep. | 7
|-1 if collection count is undefined for this collector.
|
+|gc-PSMarkSweep-collectionTime |approximate accumulated
collection elapsed time in milliseconds. | 914
|-1 if collection count is undefined for this collector.
|
+|gc-PSMarkSweep-name |Name of the collector.
| PS MarkSweep
|Constant value.
|
+|gc-PSScavenge-collectionCount |Total number of
collections that have occurred for PS Scavenge. | 25
|-1 if collection count is undefined for this collector.
|
+|gc-PSScavenge-collectionTime |approximate accumulated
collection elapsed time in milliseconds. | 1206
|-1 if collection count is undefined for this collector.
|
+|gc-PSScavenge-name |Name of the collector.
| PS Scavenge
|Constant value.
|
+|heapMemoryCommitted |Amount of memory in
bytes guaranteed for the JVM to use. | 375914496
|
|
+|heapMemoryMax |Max amount of memory in
bytes that can be used | 11453595648
|Also refer to spark.memory.fraction for the fraction managed by spark
|
+|heapMemoryTotalUsed |Total heap memory used
in bytes | 86386032
|Total memory used should be less than committed and max heap memory.
|
+|host |Hostname of the
executor | bigdata-datanode1.com
|
|
+|memoryPools-CodeCache-name |Name of the code cache.
| Code Cache
|Constant value.
|
+|memoryPools-CodeCache-peakUsageCommitted |Peak committed memory
of this pool since the JVM was started | 2555904
|
|
+|memoryPools-CodeCache-peakUsageMax |Peak value of usageMax
of this pool since the JVM was started | 251658240
|
|
+|memoryPools-CodeCache-peakUsageUsed |Peak value of memory of
this pool used since the JVM was started.| 582656
|
|
+|memoryPools-CodeCache-type |The type of this memory
pool. | Non-heap memory
|Code cache is a non-heap memory. So this value will always be constant.
|
+|memoryPools-CodeCache-usageCommitted |Total memory in bytes
committed for this pool. | 2555904
|
|
+|memoryPools-CodeCache-usageMax |The maximum amount of
memory that can be used by this pool. | 251658240
|-1 if undefined. Not guaranteed to be available if greater than the
committed memory. |
+|memoryPools-CodeCache-usageUsed |Amount of memory used
by this pool. | 582656
|Having some code cache memory free ensures it is usable for JIT
compilation. |
+|memoryPools-CompressedClassSpace-name |Name of the compressed
class space. | Compressed Class Space
|Is constant. This space is allocated in native memory for class metadata.
|
+|memoryPools-CompressedClassSpace-peakUsageCommitted |Peak committed memory
of this pool since JVM was started. | 1572864
|
|
+|memoryPools-CompressedClassSpace-peakUsageMax |Peak value of usageMax
of this pool since the JVM was started. | 1073741824
|
|
+|memoryPools-CompressedClassSpace-peakUsageUsed |Peak value of usage of
this pool since the JVM was started. | 1389800
|
|
+|memoryPools-CompressedClassSpace-type |Type of memory of this
memory pool. | Non-heap memory
|Compressed class space is native memory but is managed by the JVM
|
+|memoryPools-CompressedClassSpace-usageCommitted |Total memory in bytes
committed for this pool. | 1572864
|
|
+|memoryPools-CompressedClassSpace-usageMax |The maximum amount of
memory that can be used by this pool | 1073741824
|-1 if undefined. Not guaranteed to be available if greater than the
committed memory. |
+|memoryPools-CompressedClassSpace-usageUsed |Amount of memory used
by this pool. | 1389800
|
|
+|memoryPools-Metaspace-name |Name of the Metaspace.
| Metaspace
|Is constant. Metaspace is always called metaspace.
|
+|memoryPools-Metaspace-peakUsageCommitted |Peak committed memory
of this pool since JVM was started. | 12320768
|
|
+|memoryPools-Metaspace-peakUsageMax |Peak value of usageMax
of this pool since the JVM was started. | -1
|-1 if undefined.
|
+|memoryPools-Metaspace-peakUsageUsed |Peak value of usage of
this pool since the JVM was started. | 11768752
|
|
+|memoryPools-Metaspace-type |Type of memory of this
memory pool. | Non-heap memory
|Metaspace is native memory (since JDK 8) but is managed by the JVM
|
+|memoryPools-Metaspace-usageCommitted |Total memory in bytes
committed for this pool. | 12320768
|
|
+|memoryPools-Metaspace-usageMax |The maximum amount of
memory that can be used by this pool | -1
|-1 if undefined. Not guaranteed to be available if greater than the
committed memory. |
+|memoryPools-Metaspace-usageUsed |Amount of memory used
by this pool. | 11768752
|The amount of space used by loaded classes.
|
+|memoryPools-PSEdenSpace-name |Name of the young
generation heap space. | PS Eden Space
|Is constant. PS Eden space is always called PS Eden Space.
|
+|memoryPools-PSEdenSpace-peakUsageCommitted |Peak committed memory
of this pool since JVM was started. | 98041856
|
|
+|memoryPools-PSEdenSpace-peakUsageMax |Peak value of usageMax
of this pool since the JVM was started. | 4262461440
|-1 if undefined.
|
+|memoryPools-PSEdenSpace-peakUsageUsed |Peak value of usage of
this pool since the JVM was started. | 92270136
|
|
+|memoryPools-PSEdenSpace-type |Type of memory of this
memory pool. | Heap memory
|PS Eden space is a portion of the heap memory that is managed by the JVM
|
+|memoryPools-PSEdenSpace-usageCommitted |Total memory in bytes
committed for this pool. | 98041856
|
|
+|memoryPools-PSEdenSpace-usageMax |The maximum amount of
memory that can be used by this pool | 8589934592
|-1 if undefined. Not guaranteed to be available if greater than the
committed memory. |
+|memoryPools-PSEdenSpace-usageUsed |Amount of memory used
by this pool (and by newly loaded classes).| 92270136
|A continuously increasing value indicates an unhealthy garbage collection.
|
+|memoryPools-PSOldGen-name |Name of the old
generation heap space. | PS Old Gen
|Is constant. PS Old Gen space is always called PS Old Gen.
|
+|memoryPools-PSOldGen-peakUsageCommitted |Peak committed memory
of this pool since JVM was started. | 261619712
|
|
+|memoryPools-PSOldGen-peakUsageMax |Peak value of usageMax
of this pool since the JVM was started. | 8589934592
|-1 if undefined.
|
+|memoryPools-PSOldGen-peakUsageUsed |Peak value of usage of
this pool since the JVM was started. | 0
|
|
+|memoryPools-PSOldGen-type |Type of memory of this
memory pool. | Heap memory
|PS Old Gen space is a portion of the heap memory that is managed by the JVM
|
+|memoryPools-PSOldGen-usageCommitted |Total memory in bytes
committed for this pool. | 261619712
|
|
+|memoryPools-PSOldGen-usageMax |The maximum amount of
memory that can be used by this pool | 8589934592
|-1 if undefined. Not guaranteed to be available if greater than the
committed memory. |
+|memoryPools-PSOldGen-usageUsed |Amount of memory used
by this pool (by objects beyond threshold).| 0
|A continuously increasing value indicates an unhealthy garbage collection.
|
+|memoryPools-PSSurvivorSpace-name |Name of the survivor
heap space. | PS Survivor Space
|Is constant. PS Survivor Space is always called PS Survivor Space.
|
+|memoryPools-PSSurvivorSpace-peakUsageCommitted |Peak committed memory
of this pool since JVM was started. | 16252928
|
|
+|memoryPools-PSSurvivorSpace-peakUsageMax |Peak value of usageMax
of this pool since the JVM was started. | 16252928
|-1 if undefined.
|
+|memoryPools-PSSurvivorSpace-peakUsageUsed |Peak value of usage of
this pool since the JVM was started. | 0
|
|
+|memoryPools-PSSurvivorSpace-type |Type of memory of this
memory pool. | Heap memory
|PS Survivor Space is a portion of the heap memory that is managed by the JVM
|
+|memoryPools-PSSurvivorSpace-usageCommitted |Total memory in bytes
committed for this pool. | 16252928
|
|
+|memoryPools-PSSurvivorSpace-usageMax |The maximum amount of
memory that can be used by this pool | 16252928
|-1 if undefined. Not guaranteed to be available if greater than the
committed memory. |
+|memoryPools-PSSurvivorSpace-usageUsed |Amount of memory used
by this pool (and by newly loaded classes).| 0
|A continuously increasing value indicates an unhealthy garbage collection.
|
+|name |Process name (number)
| [email protected]
|Also contains host name.
|
+|nonHeapMemoryCommitted |Amount of non heap
memory in bytes committed for the JVM to use. | 16187392
|Total of committed of all non-heap memory pools.
|
+|nonHeapMemoryMax |Maximum amount of
memory in bytes that can be used. | -1
|-1 if undefined.
|
+|nonHeapMemoryTotalUsed |Amount of non heap
memory (metaspace, code cache) used. | 13510368
|Total of usage of all non-heap memory pools.
|
+|processCpuLoad |The recent cpu usage
for the JVM process. | 4.335671366438364e-9
|
|
+|processCpuTime |CPU time used by JVM
process in nanoseconds. | 2340000000
|
|
+|processUuid |UUID of the JVM
process. |
48abf05b-dfd4-4069-86c6-8cdbc195bd40 |
|
+|processUuid_1 |UUID of the JVM
process. |
48abf05b-dfd4-4069-86c6-8cdbc195bd40 |Appears to be a duplicate metric.
|
+|role |Role of this spark
executor. | executor
|Can be either executor or driver.
|
+|systemCpuLoad |The recent cpu usage
for the whole system. | 0.03703887412577616
|
|
+|vmHWM |Peak resident set size
("High Water Mark") in bytes for the JVM. | 142876672
|An indication of actual physical memory (RAM) usage in linux by the JVM.
|
+|vmPeak |Peak of the virtual
memory usage in bytes for the JVM. | 15130554368
|
|
+|vmRSS |JVM physical memory
usage (resident set size) in bytes. | 142876672
|
|
+|vmSize |Virtual memory size for
the JVM. | 15130550272
|This memory includes swap space, so a large value can imply high swap memory
usage. |
+
+## 2. IO
+
+|**Name** |**Description**
| **Sample Value** |**Notes**
|
+|-----------------------|-----------------------------------------------------------------|---------------------------------------|---------------------------------------------------------------------------------------|
+|time |Time series value for grafana database.
| 1582504713009000000 |
|
+|appId |Application id for the job that is running.
| application_1576797426048_311346 |If using yarn, this is the
yarn application id. |
+|epochMillis |System time in milliseconds when this value was
recorded. | 1582504713001 |
|
+|host |Hostname of the executor.
| bigdata-datanode1.com |
|
+|name |Process name (number).
| [email protected] |Includes the hostname as
well. |
+|processUuid |UUID of the JVM process.
| 89a256c7-96c3-4e48-b5ba-f55e886ba743 |
|
+|processUuid_1 |UUID of the JVM process.
| 89a256c7-96c3-4e48-b5ba-f55e886ba743 |Appears to be a duplicate
metric. |
+|role |Role of this spark executor.
| executor |Can be either executor or
driver. |
+|self-io-rchar-1 |The number of characters read by this JVM process.
| 8192340 |The "-1" is coming from the
formatter for influx which was used when writting this. |
+|self-io-read_bytes-1 |Attempt to count the bytes read by this JVM process
from storage.| 1892352 |
|
+|self-io-wchar-1 |The number of characters written to disk by this JVM
process. | 51016 |
|
+|self-io-write_bytes-1 |Attempt to count the bytes written by JVM process to
storage. | 86016 |
|
+|stat-cpu-N |CPU entry found in /proc/<pid>/stat.
| cpu, cpu0, cpu1, cpu2 ... |N is the CPU number
starting at cpu-1 (i.e. cpu) which aggregates all other CPU lines. |
+|stat-idle-N |Time the N-th CPU spent in the idle task.
| 53679018 |Time is in USER_HZ (1/100th
of a second). |
+|stat-iowait-N |Time spent by N-th CPU waiting for I/O to complete.
| 137645010 |
|
+|stat-nice-N |Time spent by N-th CPU in user mode with low priority
(nice). | 104 |Low idle time with high
nice time are indications of busy background tasks running. |
+|stat-system-N |Time spent by N-th CPU in system mode.
| 32700364 |Time spent at kernel level.
|
+|stat-user-N |Time spent by N-th CPU at the user (application)
level | 279816973 |
|
+
+## 3. Process Info
+
+|**Name** |**Description**
| **Sample Value** |**Notes**
|
+|------------------|-----------------------------------------------------------------|------------------------------------------------|---------------------------------------------------------------------------------------|
+|time |Time series value for grafana database.
| 1582504713009000000 |
|
+|agentVersion |A constant value defined in code.
| 1.0.0 |Agent version will
always be 1.0.0. |
+|appId |Application id for the job that is running.
| application_1576797426048_311346 |If using yarn, this is
the yarn application id. |
+|cmdline |Contains JVM command executed and input arguments and
classpath. | /usr/jdk64/jdk1.8.0_112/bin/java -javaagent... |
|
+|epochMillis |System time in milliseconds when this value was recorded.
| 1582504713001 |
|
+|host |Hostname of the executor.
| bigdata-datanode1.com |
|
+|jvmClassPath |Contains JVM classpath entries.
| --user-class-path .../jvm-profiler-1.0.0.jar |This entry is emtpy if
cmdLine is not empty. This is to avoid duplicate information. |
+|jvmInputArguments |Contains JVM input arguments.
| metricInterval=5000,sampleInterval=5000 ... |This entry is emtpy if
cmdLine is not empty. This is to avoid duplicate information. |
+|name |Process name (number).
| [email protected] |Includes the hostname
as well. |
+|processUuid |UUID of the JVM process.
| 89a256c7-96c3-4e48-b5ba-f55e886ba743 |
|
+|processUuid_1 |UUID of the JVM process.
| 89a256c7-96c3-4e48-b5ba-f55e886ba743 |Appears to be a
duplicate metric. |
+|role |Role of this spark executor.
| executor |Can be either executor
or driver. |
+|xmxBytes |Maximum JVM heap size in bytes.
| 12884901888 |
|
+
+## 4. Stack Trace
+
+|**Name** |**Description**
| **Sample Value** |**Notes**
|
+|--------------|-----------------------------------------------------------------|-----------------------------------------|---------------------------------------------------------------------------------------|
+|time |Time series value for grafana database.
| 1582504713009000000 |
|
+|appId |Application id for the job that is running.
| application_1576797426048_311346 |If using yarn, this is the yarn
application id. |
+|count |Number of elements appended to the metrics buffer.
| 1 |This value is usually 1, but not
guaranteed to be constant. It is thread-safe. |
+|endEpoch |System time in milliseconds after resetting the metrics
buffer. | 1582504712723 |The metrics buffer is an
internal buffer used by this profiling application. |
+|host |Hostname of the executor.
| bigdata-datanode1.com |
|
+|name |Process name (number).
| [email protected] |Includes the hostname as well.
|
+|processUuid |UUID of the JVM process.
| 89a256c7-96c3-4e48-b5ba-f55e886ba743 |
|
+|processUuid_1 |UUID of the JVM process.
| 89a256c7-96c3-4e48-b5ba-f55e886ba743 |Appears to be a duplicate metric.
|
+|role |Role of this spark executor.
| executor |Can be either executor or driver.
|
+|stacktrace |Stacktrace starting from bottom of the stacktrace.
| java.lang.Object.wait,java.lang.ref ... |Trims the top method (most nested
method) if the size is too large (800k characters) |
+|startEpoch |System time in milliseconds before resetting the metrics
buffer. | 1582504710523 |The metrics buffer is an
internal buffer used by this profiling application. |
+|threadName |Name of the thread for which stacktrace is displayed.
| Signal Dispatcher |
|
+|threadState |State of the thread when this stacktrace was captured.
| NEW, RUNNABLE, BLOCKED, WAITING ... |
|
+
+## 5. Thread Info
+
+|**Name** |**Description**
| **Sample Value** |**Notes**
|
+|------------------------|----------------------------------------------------------|---------------------------------------|------------------------------------------------|
+|time |Time series value for grafana database.
| 1582504713009000000 |
|
+|epochMillis |System time in milliseconds when this value was
recorded. | 1582504781508 |
|
+|host |Hostname of the executor.
| bigdata-datanode1.com |
|
+|name |Process name (number).
| [email protected] |Includes the hostname as well.
|
+|appId |Application id for the job that is running.
| application_1576797426048_311346 |If using yarn, this is the yarn
application id. |
+|processUuid |UUID of the JVM process.
| 89a256c7-96c3-4e48-b5ba-f55e886ba743 |
|
+|role |Role of this spark executor.
| executor |Can be either executor or driver.
|
+|totalStartedThreadCount |Total Number of Thread created so far since JVm
Launch. | 10 |
|
+|liveThreadCount |Number of thread which are currently active.
| 8 |
|
+|newThreadCount |Number of thread which are newly Created.
| 3 | Derived from last Cycle Total
Threads. |
+|peakThreadCount |the peak live thread count since the JVM start
| 8 |
|
