[
https://issues.apache.org/jira/browse/KYLIN-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089245#comment-16089245
]
Zhong Yanghong commented on KYLIN-2721:
---------------------------------------
Hi [[email protected]],
bq. Kylin is already using {{org.apache.hadoop.metrics2}} in {{QueryMetrics}}.
How necessary moving to a new metrics framework?
The philosophy of this metrics framework [^Metrics Framework.png] is different
from the one of {{QueryMetrics}}. This metrics framework does not maintain the
metrics in memory. It's mainly for collecting metrics from *multiple kylin
servers* and reporting them to multiple storages acting as the source of system
cubes which can serve queries. After enabling kylin streaming for those system
cubes, metrics can be shown by real time.
bq. Assume we agree on moving to the new metrics framework, we will also
refactor the current {{QueryMetrics}} such that Kylin don't end up with two
separate metrics framework, right?
As I remember, the previous hadoop metrics is not so stable. Thus we choose
another lightweight metrics framework, yammer metrics which is widely used by
others, like kafka.
This metrics framework is an extend to rather than heavily depends on yammer
metrics. It can also be an extend to hadoop metrics2. We may need to decide
which one to choose.
bq. Regarding collecting system metrics, like CPU, Memory etc, there are plenty
of tools already. I don't think Kylin should re-invent the wheel and do it
again. Does KYLIN-2724 mean something else, like integrate with those existing
tools?
For system metrics, we made an extension by introducing collecting frequency.
Since collecting all of system metrics too frequently may bring heavy workload
to servers, and different kind of metrics owns different priority, we divide
system metrics into three groups with different collecting frequencies as
follows:
* high frequency metrics(cpuUsage, jvmCpuUsage, jvmMemAvail, javaFDs, sysFDs)
* medium frequency metrics(jvmMemTotal, jvmVirtualBytes)
* low frequency metrics(avgSystemLoad, serverUpTimeInHrs, physMemAvail,
physMemTotal, jvmMemUsed, jvmPrivateBytes, maxJavaFDs, maxSysFDs)
We may leverage whether we can do this extension based on existing tools
further.
bq. Could you elaborate more on the system cube?
I'll do this with my patch soon.
> Introduce a new metrics framework based on yammer metrics
> ---------------------------------------------------------
>
> Key: KYLIN-2721
> URL: https://issues.apache.org/jira/browse/KYLIN-2721
> Project: Kylin
> Issue Type: New Feature
> Affects Versions: v2.0.0
> Reporter: Zhong Yanghong
> Assignee: Zhong Yanghong
> Attachments: Metrics Framework.png
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
