Clarification from Amith:
Hi Nathaniel,
Thanks a lot for your quick reply, what is confusing for me here is how
morgoth calculated anomalyScore field whose value has turned out to be
0.9897172236503856. And how is this being used to detect anomaly.
How does this particular node function
…
@morgoth()
.field(field)
.scoreField(scoreField)
.minSupport(minSupport)
.errorTolerance(errorTolerance)
.consensus(consensus)
// Configure a single Sigma fingerprinter
.sigma(sigmas).
You can choose some arbitrary data to help me understand this. :)
Thanks,
Amith
My response:
The `anomalyScore` is `1 - averageSupport`, where averageSupport is the
average of the support values returned from each or the fingerprinters. In
your case you only have one fingerprinter `sigma` so using the anomalyScore
of ~ `0.99` we can determine that the sigma fingerprinter returned a
support of ~ `0.01`. Support is defined as `count / total`, where count is
the number of times a specific event has been seen and total is the total
number events seen. The support can be interpreted as a frequency
percentage, i.e. the most recent window has only been seen 1% of the time.
Since 0.01 is < 0.05 (the min support defined) an anomaly was triggered.
Taking this back to the anomaly score it can be interpreted that 99% of the
time we do not see an event like this one.
Remember that Morgoth distinguishs different windows as different events
using the fingerprinters. In your case the sigma function is computing the
std deviation and mean of the windows it receives. If a window arrives that
is more than 3 stddevs away from the mean than it is not considered the
same event and is a unique event.
Taking all of that and putting it together receiving an anomaly score of
99% out of Morgoth for your setup can be interpreted as: You have sent
several 1m windows to Morgoth. The window that triggered the anomaly event
is only similar to ~1% of those windows, where similar is defined as being
within 3 std deviations.
On Thursday, October 27, 2016 at 9:30:13 AM UTC-6, [email protected]
wrote:
>
>
> In short there are two parts to Morgoth.
>
> 1. A system that counts the frequency of different kinds of events. This
> is the lossy counting part
> 2. A system that determines if a window of data is the same as an existing
> event being tracked or something new. This is the fingerprinting part.
>
> Here is a quick read through for those concepts
> http://docs.morgoth.io/docs/detection_framework/
>
> Its a little hard to tell if Morgoth has done anything unexpected without
> more detail. Can you share some of the data that lead to this alert, so I
> can talk to the specifics of what is going on? Or maybe you could ask a
> more specific question about which part is confusing?
>
>
>
> On Thursday, October 27, 2016 at 6:47:02 AM UTC-6, [email protected]
> wrote:
>>
>> Hi All,
>>
>> I am trying to run morgoth as a child process to kapacitor, but I am
>> failing understand how morgoth functions. Below is the sample tick script I
>> tried out of the Morgoth docs. This is generating some alerts but I am
>> unable to figure out if they are suppose to get triggered way they have.
>> Pasting a snippet out of alert as well.
>>
>> I basically want to understand the functioning of Morgoth through this
>> example.
>>
>> Alert
>> ===================================================================
>>
>> {
>> "id":"cpu:cpu=cpu-total,host=ip-10-121-48-24.ec2.internal,",
>> "message":"cpu:cpu=cpu-total,host=ip-10-121-48-24.ec2.internal, is
>> CRITICAL",
>> "details":"",
>> "time":"2016-10-27T11:33:00Z",
>> "duration":21780000000000,
>> "level":"CRITICAL",
>> "data":{
>> "series":[
>> {
>> "name":"cpu",
>> "tags":{
>> "cpu":"cpu-total",
>> "host":"ip-10-121-48-24.ec2.internal"
>> },
>> "columns":[
>> "time",
>> "anomalyScore",
>> "usage_guest",
>> "usage_guest_nice",
>> "usage_idle",
>> "usage_iowait",
>> "usage_irq",
>> "usage_nice",
>> "usage_softirq",
>> "usage_steal",
>> "usage_system",
>> "usage_user"
>> ],
>> "values":[
>> [
>> "2016-10-27T11:33:00Z",
>> 0.9897172236503856,
>> 0,
>> 0,
>> 99.49748743708487,
>> 0,
>> 0,
>> 0,
>> 0,
>> 0,
>> 0.5025125628122904,
>> 0
>> ]
>>
>> ===================================================================
>> // The measurement to analyze
>> var measurement = 'cpu'
>>
>> // Optional group by dimensions
>> var groups = [*]
>>
>> // Optional where filter
>> var whereFilter = lambda: TRUE
>>
>> // The amount of data to window at once
>> var window = 1m
>>
>> // The field to process
>> var field = 'usage_idle'
>>
>> // The name for the anomaly score field
>> var scoreField = 'anomalyScore'
>>
>> // The minimum support
>> var minSupport = 0.05
>>
>> // The error tolerance
>> var errorTolerance = 0.01
>>
>> // The consensus
>> var consensus = 0.5
>>
>> // Number of sigmas allowed for normal window deviation
>> var sigmas = 3.0
>>
>> stream
>> // Select the data we want
>> |from()
>> .measurement(measurement)
>> .groupBy(groups)
>> .where(whereFilter)
>> // Window the data for a certain amount of time
>> |window()
>> .period(window)
>> .every(window)
>> .align()
>> // Send each window to Morgoth
>> @morgoth()
>> .field(field)
>> .scoreField(scoreField)
>> .minSupport(minSupport)
>> .errorTolerance(errorTolerance)
>> .consensus(consensus)
>> // Configure a single Sigma fingerprinter
>> .sigma(sigmas)
>> // Morgoth returns any anomalous windows
>> |alert()
>> .details('')
>> .crit(lamda: TRUE)
>> .log('/tmp/cpu_alert.log')
>>
>>
--
Remember to include the version number!
---
You received this message because you are subscribed to the Google Groups
"InfluxData" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/influxdb.
To view this discussion on the web visit
https://groups.google.com/d/msgid/influxdb/12cd2956-b0f3-4664-850e-ce54dc721cae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
