[
https://issues.apache.org/jira/browse/EAGLE-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhao, Qingwen updated EAGLE-1041:
---------------------------------
Description:
In some cases, like increment or decrement pattern, data need to be processed
in more than one stage. For example, alert if the metric value increases by N.
Two steps to get the right alert.
1. sort & filter events which meet the filter conditions.
2. define data change pattern.
Here is a sample policy
{code}
fromHADOOP_JMX_METRIC_STREAM_SANDBOX[metric=="hadoop.namenode.dfs.missingblocks"]#window.externalTime(timestamp,1min)
select * group by site,host,component, metric insert into temp;
from every
a=HADOOP_JMX_METRIC_STREAM_SANDBOX[metric=="hadoop.namenode.dfs.missingblocks"],
b=HADOOP_JMX_METRIC_STREAM_SANDBOX[b.component==a.component and
b.metric==a.metric and b.host==a.host and b.value>a.value and a.value>100]
select b.site,b.host,b.component, b.metric, b.value as newNumOfMissingBlocks,
a.value as oldNumOfMissingBlocks, (b.value-a.value) as
increastedNumOfMissingBlocks insert into
HADOOP_JMX_METRIC_STREAM_SANDBOX_MISS_BLOCKS_LARGER_OUT;
{code}
There are two queries in this policy. The first one with the time window
condition tells Eagle to sort the original events. The second one defines the
data pattern. As the constraint of Siddhi syntax
(https://docs.wso2.com/display/CEP420/SiddhiQL+Guide+3.1#SiddhiQLGuide3.1-Pattern),
the filtering of input events does not work.
Luckily, if we put the output stream of the first query as the input stream of
the second query, it works. That's the problem this ticket tries to solve.
Ideally, the right policy can be written as
{code}
fromHADOOP_JMX_METRIC_STREAM_SANDBOX[metric=="hadoop.namenode.dfs.missingblocks"]#window.externalTime(timestamp,1min)
select * group by site,host,component, metric insert into MISSING_BLOCK_OUT;
from every a=MISSING_BLOCK_OUT[metric=="hadoop.namenode.dfs.missingblocks"],
b=MISSING_BLOCK_OUT[b.component==a.component and b.metric==a.metric and
b.host==a.host and b.value>a.value and a.value>100]
select b.site,b.host,b.component, b.metric, b.value as newNumOfMissingBlocks,
a.value as oldNumOfMissingBlocks, (b.value-a.value) as
increastedNumOfMissingBlocks insert into
HADOOP_JMX_METRIC_STREAM_SANDBOX_MISS_BLOCKS_LARGER_OUT;
{code}
was:
In some cases, like increment or decrement pattern, data need to be processed
in more than one stage. For example, alert if the metric value increases by N.
Two steps to get the right alert.
1. sort & filter events which meet the filter conditions.
2. define data change pattern.
Here is a sample policy
{code}
fromHADOOP_JMX_METRIC_STREAM_SANDBOX[metric=="hadoop.namenode.dfs.missingblocks"]#window.externalTime(timestamp,1min)
select * group by site,host,component, metric insert into temp;
from every
a=HADOOP_JMX_METRIC_STREAM_SANDBOX[metric=="hadoop.namenode.dfs.missingblocks"],
b=HADOOP_JMX_METRIC_STREAM_SANDBOX[b.component==a.componentandb.metric==a.metricandb.host==a.hostandb.value>a.valueanda.value>100]
selectb.site,b.host,b.component, b.metric, b.value as newNumOfMissingBlocks,
a.value as oldNumOfMissingBlocks, (b.value-a.value) as
increastedNumOfMissingBlocks insert into
HADOOP_JMX_METRIC_STREAM_SANDBOX_MISS_BLOCKS_LARGER_OUT;
{code}
There are two queries in this policy. The first one with the time window
condition tells Eagle to sort the original events. The second one defines the
data pattern. As the constraint of Siddhi syntax
(https://docs.wso2.com/display/CEP420/SiddhiQL+Guide+3.1#SiddhiQLGuide3.1-Pattern),
the filtering of input events does not work.
Luckily, if we put the output stream of the first query as the input stream of
the second query, it works. That's the problem this ticket tries to solve.
Ideally, the right policy can be written as
{code}
fromHADOOP_JMX_METRIC_STREAM_SANDBOX[metric=="hadoop.namenode.dfs.missingblocks"]#window.externalTime(timestamp,1min)
select * group by site,host,component, metric insert into MISSING_BLOCK_OUT;
from every a=MISSING_BLOCK_OUT[metric=="hadoop.namenode.dfs.missingblocks"],
b=MISSING_BLOCK_OUT[b.component==a.componentandb.metric==a.metricandb.host==a.hostandb.value>a.valueanda.value>100]
selectb.site,b.host,b.component, b.metric, b.value as newNumOfMissingBlocks,
a.value as oldNumOfMissingBlocks, (b.value-a.value) as
increastedNumOfMissingBlocks insert into
HADOOP_JMX_METRIC_STREAM_SANDBOX_MISS_BLOCKS_LARGER_OUT;
{code}
> Support policy processing pipeline
> ----------------------------------
>
> Key: EAGLE-1041
> URL: https://issues.apache.org/jira/browse/EAGLE-1041
> Project: Eagle
> Issue Type: Improvement
> Affects Versions: v0.5.0
> Reporter: Zhao, Qingwen
> Assignee: Zhao, Qingwen
>
> In some cases, like increment or decrement pattern, data need to be processed
> in more than one stage. For example, alert if the metric value increases by
> N. Two steps to get the right alert.
> 1. sort & filter events which meet the filter conditions.
> 2. define data change pattern.
> Here is a sample policy
> {code}
> fromHADOOP_JMX_METRIC_STREAM_SANDBOX[metric=="hadoop.namenode.dfs.missingblocks"]#window.externalTime(timestamp,1min)
> select * group by site,host,component, metric insert into temp;
> from every
> a=HADOOP_JMX_METRIC_STREAM_SANDBOX[metric=="hadoop.namenode.dfs.missingblocks"],
> b=HADOOP_JMX_METRIC_STREAM_SANDBOX[b.component==a.component and
> b.metric==a.metric and b.host==a.host and b.value>a.value and a.value>100]
> select b.site,b.host,b.component, b.metric, b.value as newNumOfMissingBlocks,
> a.value as oldNumOfMissingBlocks, (b.value-a.value) as
> increastedNumOfMissingBlocks insert into
> HADOOP_JMX_METRIC_STREAM_SANDBOX_MISS_BLOCKS_LARGER_OUT;
> {code}
> There are two queries in this policy. The first one with the time window
> condition tells Eagle to sort the original events. The second one defines the
> data pattern. As the constraint of Siddhi syntax
> (https://docs.wso2.com/display/CEP420/SiddhiQL+Guide+3.1#SiddhiQLGuide3.1-Pattern),
> the filtering of input events does not work.
> Luckily, if we put the output stream of the first query as the input stream
> of the second query, it works. That's the problem this ticket tries to solve.
> Ideally, the right policy can be written as
> {code}
> fromHADOOP_JMX_METRIC_STREAM_SANDBOX[metric=="hadoop.namenode.dfs.missingblocks"]#window.externalTime(timestamp,1min)
> select * group by site,host,component, metric insert into MISSING_BLOCK_OUT;
> from every a=MISSING_BLOCK_OUT[metric=="hadoop.namenode.dfs.missingblocks"],
> b=MISSING_BLOCK_OUT[b.component==a.component and b.metric==a.metric and
> b.host==a.host and b.value>a.value and a.value>100]
> select b.site,b.host,b.component, b.metric, b.value as newNumOfMissingBlocks,
> a.value as oldNumOfMissingBlocks, (b.value-a.value) as
> increastedNumOfMissingBlocks insert into
> HADOOP_JMX_METRIC_STREAM_SANDBOX_MISS_BLOCKS_LARGER_OUT;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
