The profile periods are mergeable so an event that occurs over several profile periods can be detected. The profiler flushes according the specified time periods. The idea is to build a historical baseline and compare against it. What you are proposing is a deterministic rule, which we are trying to de-emphasize in the system. The Metron way of doing "number of ssh connection over the last x minutes bigger than N" would be "take N number of connections I have during my current time period, compare this value against the historical baseline, is this value anomalous? yes -> alert, no-> do nothing. There is no set threshold to compare against, which makes the system more robust.
I am not sure I currently see a reason to flush profiles based on specific events. Someone else may have a different view, though.
Thanks,
James
26.06.2018, 15:28, "Michel Sumbul" <[email protected]>:
Hello Metron Guru,I would like to know if there's way to have a profiler that will flush the result to hbase not based on a time period but on some conditions.For example, if we are tracking a specific sequence over the time for an ip/user like (event A, event b and event C). If this entire sequence happens inside the profil period duration its fine, but if this sequence happens over 2 or more profiler periods then it will not be detected. Moreover if the sequence occured in 1 sec but the profiler period is 15 minutes, then it will wait a long time before being flush to hbase.Another use case will be, if you are looking at the average of something and a threshold is reach to directly flush it and then generate an alert asap. Like number of ssh connection over the last x minutes bigger than N, then flush it. otherwise continue to profile the user.My first question is, is it currently possible to do that, because I have not found how.Secondly if it not feasible for the moment, do you think that might be an useful feature?I was also thinking that the result might be flush to a specific kafka topic and not to hbase. For example, if the profiler detect an anomaly on the behavior like number of ssh connection, or sequence of event, it flush the result to a kafka topic with all the "real-time" alert.Maybe this already exist and profiler is not the good place to do that, to be honest I'm not sure. But I suppose that Im not the first that imagine that, so any comment on how to realize this is welcome :)
Best regard,
Michel
-------------------
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
