FWIW, We have a built a similar Log Aggregator internally using Apache Nifi
+ KFC stack

(KFC = Kafka, Flink, Cassandra)

Using Apache NiFi for ingesting logs from Openstack via rsyslog and writing
them out to Kafka topics -> Flink Streaming + CEP for detecting anomalous
patterns -> persist the patterns with relevant metadata to Cassandra ->
Dashboard or Search Engine.

We are using Flink CEP for detecting patterns in server logs and to flag
alerts onto a dashboard.

You can check out the implementation here -
https://github.com/keedio/openstack-log-processor



On Thu, Jul 20, 2017 at 5:47 PM, Branham, Jeremy [IT] <
[email protected]> wrote:

> Raj -
> I'm looking for the same thing.
> As the ML library doesn't support DataStream api, I'm tossing ideas around
> maybe using the windowing function to build up a model that changes over
> time.
>
>
>
> Jeremy D. Branham
> Technology Architect - Sprint
> O: +1 (972) 405-2970 | M: +1 (817) 791-1627
> [email protected]
> #gettingbettereveryday
>
>
> -----Original Message-----
> From: Raj Kumar [mailto:[email protected]]
> Sent: Thursday, July 20, 2017 4:24 PM
> To: [email protected]
> Subject: Flink Anomaly Detection
>
> Hi,
>
> I don't see much discussion on Anomaly detection using Flink. we are
> working on a project where we need to monitor the server logs in real time.
> If there is any sudden spike in the number of transactions(Unusual), server
> errors, we need to create an alert.
>
> 1. How can we best achieve this?
> 2. How do we store the historical information about the patterns observed
> and compute the baseline? Do we need any external source like Elasticsearch
> to store the window snapshots to build a baseline?
> 3. Baseline should be self-learning as new patterns are discovered and
> baseline should get adjusted based on this.
> 4. Flink ML has any capabilities to achieve this?
>
> Please let me know if you have any approach/suggestions ?
>
>
>
> --
> View this message in context: https://na01.safelinks.
> protection.outlook.com/?url=http%3A%2F%2Fapache-flink-
> user-mailing-list-archive.2336050.n4.nabble.com%2FFlink-
> Anomaly-Detection-tp14370.html&data=02%7C01%7CJeremy.D.
> Branham%40sprint.com%7C0fd3a0f94d3547bdf12b08d4cfb86865%
> 7C4f8bc0acbd784bf5b55f1b31301d9adf%7C0%7C0%7C636361838310440993&sdata=
> Rah8P27ro%2FT5xZJAN%2BFwQv0Ze%2FGD9WuF6lGM3ox1Mac%3D&reserved=0
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>
> ________________________________
>
> This e-mail may contain Sprint proprietary information intended for the
> sole use of the recipient(s). Any use by others is prohibited. If you are
> not the intended recipient, please contact the sender and delete all copies
> of the message.
>

Reply via email to