[
https://issues.apache.org/jira/browse/EAGLE-928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hao Chen updated EAGLE-928:
---------------------------
Description:
Refine system metric schema design and fix system metric collector
h1. Principle
Any metric (no mater sys metric) should at least have three fields: "metric",
"group", "timestamp", "value",
"group" is the metric category path separated with slash, for example "FIRST
CATEGORY.SECOND CATEGORY", which will be commonly used for well organizing
metric groups in customized metric dashboard.
h1. Stream Name for System Metric
{code}
<stream>
<streamId>SYSTEM_METRIC_STREAM</streamId>
<description>System Metrics Stream including CPU, Network, Disk,
etc.</description>
<columns>
<column>
<name>host</name>
<type>string</type>
</column>
<column>
<name>timestamp</name>
<type>long</type>
</column>
<column>
<name>metric</name>
<type>string</type>
</column>
<column>
<name>group</name>
<type>string</type>
</column>
<column>
<name>site</name>
<type>string</type>
</column>
<column>
<name>device</name>
<type>string</type>
</column>
<column>
<name>value</name>
<type>double</type>
<defaultValue>0.0</defaultValue>
</column>
</columns>
</stream>
{code}
h1. Stream Schema for System metric
* *metric*: [STRING] metric name string
* *group*: [STRING] metric group/type
* *timestamp*: [LONG] metric generation time
* *site*: [STRING] siteId
* *host*: [STRING] source host name
* *device*: [STRING] device name, like cpu
h1. Sample CPU Metric
{code}
{
"timestamp": 1487918913569,
"metric": "system.cpu.usage",
"group": "system.cpu",
"site": "sandbox",
"value": 0.058,
"host": "sandbox.hortonworks.com",
"device": "cpu7"
}
{code}
h1. Sample Network Metrics
{code}
{
"timestamp": 1487918913569,
"metric": "system.nic.transmitdrop",
"group": "system.network",
"site": "sandbox",
"value": 7724.0,
"host": "sandbox.hortonworks.com",
"device": "eth0"
}
{code}
h1. Sample Metric Schema
{code}
{
"tags": {
"site": "sandbox",
"name": "system.memory.nfs_unstable.kb",
"group": "system.memory"
},
"dimensionFields": [
"host",
"group",
"site",
"device"
],
"metricFields": [
"value"
],
"granularity": "MINUTE",
"modifiedTimestamp": 1488190388479
}
{code}
h1. Stream Persist Topology
{code}
environment.newApp(config)
.fromStream("HADOOP_JMX_METRIC_STREAM")
.saveAsMetric(
MetricDescriptor.metricGroupAs((MetricGroupSelector)
event -> {
if (event.containsKey("component")) {
return String.format("hadoop.%s", ((String)
event.get("component")).toLowerCase());
} else {
return "hadoop.metrics";
}
})
.siteByField("site")
.namedByField("metric")
.eventTimeByField("timestamp")
.dimensionFields("host", "component", "site")
.granularity(Calendar.MINUTE)
.valueField("value"))
.fromStream("SYSTEM_METRIC_STREAM")
.saveAsMetric(MetricDescriptor.metricGroupByField("group")
.siteByField("site")
.namedByField("metric")
.eventTimeByField("timestamp")
.dimensionFields("host", "group", "site", "device")
.granularity(Calendar.MINUTE)
.valueField("value")
)
.toTopology();
{code}
was:
Refine system metric schema design and fix system metric collector
h1. Principle
Any metric (no mater sys metric) should at least have three fields: "metric",
"group", "timestamp", "value",
"group" is the metric category path separated with slash, for example "FIRST
CATEGORY.SECOND CATEGORY", which will be commonly used for well organizing
metric groups in customized metric dashboard.
h1. Stream Name for System Metric
{code}
<stream>
<streamId>SYSTEM_METRIC_STREAM</streamId>
<description>System Metrics Stream including CPU, Network, Disk,
etc.</description>
<columns>
<column>
<name>host</name>
<type>string</type>
</column>
<column>
<name>timestamp</name>
<type>long</type>
</column>
<column>
<name>metric</name>
<type>string</type>
</column>
<column>
<name>group</name>
<type>string</type>
</column>
<column>
<name>site</name>
<type>string</type>
</column>
<column>
<name>device</name>
<type>string</type>
</column>
<column>
<name>value</name>
<type>double</type>
<defaultValue>0.0</defaultValue>
</column>
</columns>
</stream>
{code}
h1. Stream Schema for System metric
* *metric*: [STRING] metric name string
* *group*: [STRING] metric group/type
* *timestamp*: [LONG] metric generation time
* *site*: [STRING] siteId
* *host*: [STRING] source host name
* *device*: [STRING] device name, like cpu
h1. Sample CPU Metric
{code}
{
"timestamp": 1487918913569,
"metric": "system.cpu.usage",
"group": "system.cpu",
"site": "sandbox",
"value": 0.058,
"host": "sandbox.hortonworks.com",
"device": "cpu7"
}
{code}
h1. Sample Network Metrics
{code}
{
"timestamp": 1487918913569,
"metric": "system.nic.transmitdrop",
"group": "system.network",
"site": "sandbox",
"value": 7724.0,
"host": "sandbox.hortonworks.com",
"device": "eth0"
}
{code}
h1. Sample Metric Schema
{code}
{
"tags": {
"site": "sandbox",
"name": "system.memory.nfs_unstable.kb",
"group": "system.memory"
},
"dimensionFields": [
"host",
"group",
"site",
"device"
],
"metricFields": [
"value"
],
"granularity": "MINUTE",
"modifiedTimestamp": 1488190388479
}
{code}
> Refine system metric schema design and fix system metric collector
> ------------------------------------------------------------------
>
> Key: EAGLE-928
> URL: https://issues.apache.org/jira/browse/EAGLE-928
> Project: Eagle
> Issue Type: Bug
> Affects Versions: v0.5.0
> Reporter: Hao Chen
> Assignee: Hao Chen
> Fix For: v0.5.0
>
>
> Refine system metric schema design and fix system metric collector
> h1. Principle
> Any metric (no mater sys metric) should at least have three fields: "metric",
> "group", "timestamp", "value",
> "group" is the metric category path separated with slash, for example "FIRST
> CATEGORY.SECOND CATEGORY", which will be commonly used for well organizing
> metric groups in customized metric dashboard.
> h1. Stream Name for System Metric
> {code}
> <stream>
> <streamId>SYSTEM_METRIC_STREAM</streamId>
> <description>System Metrics Stream including CPU, Network, Disk,
> etc.</description>
> <columns>
> <column>
> <name>host</name>
> <type>string</type>
> </column>
> <column>
> <name>timestamp</name>
> <type>long</type>
> </column>
> <column>
> <name>metric</name>
> <type>string</type>
> </column>
> <column>
> <name>group</name>
> <type>string</type>
> </column>
> <column>
> <name>site</name>
> <type>string</type>
> </column>
> <column>
> <name>device</name>
> <type>string</type>
> </column>
> <column>
> <name>value</name>
> <type>double</type>
> <defaultValue>0.0</defaultValue>
> </column>
> </columns>
> </stream>
> {code}
> h1. Stream Schema for System metric
>
> * *metric*: [STRING] metric name string
> * *group*: [STRING] metric group/type
> * *timestamp*: [LONG] metric generation time
> * *site*: [STRING] siteId
> * *host*: [STRING] source host name
> * *device*: [STRING] device name, like cpu
> h1. Sample CPU Metric
> {code}
> {
> "timestamp": 1487918913569,
> "metric": "system.cpu.usage",
> "group": "system.cpu",
> "site": "sandbox",
> "value": 0.058,
> "host": "sandbox.hortonworks.com",
> "device": "cpu7"
> }
> {code}
> h1. Sample Network Metrics
> {code}
> {
> "timestamp": 1487918913569,
> "metric": "system.nic.transmitdrop",
> "group": "system.network",
> "site": "sandbox",
> "value": 7724.0,
> "host": "sandbox.hortonworks.com",
> "device": "eth0"
> }
> {code}
> h1. Sample Metric Schema
> {code}
> {
> "tags": {
> "site": "sandbox",
> "name": "system.memory.nfs_unstable.kb",
> "group": "system.memory"
> },
> "dimensionFields": [
> "host",
> "group",
> "site",
> "device"
> ],
> "metricFields": [
> "value"
> ],
> "granularity": "MINUTE",
> "modifiedTimestamp": 1488190388479
> }
> {code}
> h1. Stream Persist Topology
> {code}
> environment.newApp(config)
> .fromStream("HADOOP_JMX_METRIC_STREAM")
> .saveAsMetric(
> MetricDescriptor.metricGroupAs((MetricGroupSelector)
> event -> {
> if (event.containsKey("component")) {
> return String.format("hadoop.%s", ((String)
> event.get("component")).toLowerCase());
> } else {
> return "hadoop.metrics";
> }
> })
> .siteByField("site")
> .namedByField("metric")
> .eventTimeByField("timestamp")
> .dimensionFields("host", "component", "site")
> .granularity(Calendar.MINUTE)
> .valueField("value"))
> .fromStream("SYSTEM_METRIC_STREAM")
> .saveAsMetric(MetricDescriptor.metricGroupByField("group")
> .siteByField("site")
> .namedByField("metric")
> .eventTimeByField("timestamp")
> .dimensionFields("host", "group", "site", "device")
> .granularity(Calendar.MINUTE)
> .valueField("value")
> )
> .toTopology();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
