I like it, Otto.

I see the recipe idea implemented as a collection of GUI wizards.  The user
can login to a web interface and choose from a collection of recipes.  The
user interacts with a recipe via a GUI wizard-like mechanism.  The wizard
gathers the input needed from the user to implement a recipe.  The recipe
itself then takes that input and creates all the ugly JSON and Stellar
configuration required to actually implement the recipe in Metron.

These recipes would be great way for new users to hit the ground running
with Metron.  Advanced users could still get under-the-hood and create
their own Profiles, Triage rules, etc using JSON and Stellar without using
these recipes.

We'd also want to make the recipe infrastructure pluggable.  We want the
community to be able to quickly and easily create new recipes without
having to build a custom HTML/JS wizard page for each recipe.




On Wed, Aug 9, 2017 at 4:44 PM Casey Stella <ceste...@gmail.com> wrote:

> Well, we need that too :)  What're you thinking, procedures for stellar?
>
> On Wed, Aug 9, 2017 at 4:42 PM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> > What we need, is a way to package up some ‘recipes’ for stellar.
> > If many people are going to do this operation, then a more friendly set
> of
> > facade functions,
> > or some thing would work.
> >
> >
> >
> > On August 9, 2017 at 16:38:48, Casey Stella (ceste...@gmail.com) wrote:
> >
> > Yeah, I'm leaning toward STATS_ADD or STATS_INIT taking a list of
> numbers.
> > STATS_MERGE seems confusing.
> >
> > On Wed, Aug 9, 2017 at 4:37 PM, Nick Allen <n...@nickallen.org> wrote:
> >
> > > Or even change the behavior of STATS_MERGE, too? If STATS_MERGE gets
> raw
> > > numbers, it wraps those in a Stats object, then returns it. Then Dima's
> > > example would just work as-is.
> > >
> > > I'm not sure I like that though. Maybe so flexible as to be confusing?
> > > Thought I would throw it out as an alternative to consider.
> > >
> > >
> > >
> > >
> > > On Wed, Aug 9, 2017 at 4:31 PM Nick Allen <n...@nickallen.org> wrote:
> > >
> > > > Oh yeah, duh. Now I'm with you. That would be a good quick hit.
> > > >
> > > > The current behavior is a little nutty. If there is a list, it only
> > > > consumes the first element in the list. I'd expect that it should
> > either
> > > > do what you describe or complain that it doesn't know how to handle a
> > > > list. Easy fix though.
> > > >
> > > > [Stellar]>>> STATS_MEAN(STATS_ADD(null, 1, 2, 3))
> > > > 2.0
> > > >
> > > > [Stellar]>>> STATS_MEAN(STATS_ADD(null, [1,2,3]))
> > > > 1.0
> > > >
> > > > [Stellar]>>> STATS_COUNT(STATS_ADD(null, [1,2,3]))
> > > > 1.0
> > > >
> > > > On Wed, Aug 9, 2017 at 4:17 PM Casey Stella <ceste...@gmail.com>
> > wrote:
> > > >
> > > >> outcoming is still a HLLP object, not a statistics object, so doing
> a
> > > >> STATS_MERGE on a bunch of them wouldn't work either.
> > > >>
> > > >> On Wed, Aug 9, 2017 at 4:15 PM, Nick Allen <n...@nickallen.org>
> > wrote:
> > > >>
> > > >> > That is another problem. Isn't the simplest answer, to just change
> > > >> this...
> > > >> >
> > > >> > "result": "HLLP_CARDINALITY(outcoming)"
> > > >> >
> > > >> > to this...
> > > >> >
> > > >> > "result": "outcoming"
> > > >> >
> > > >> > ?
> > > >> >
> > > >> > On Wed, Aug 9, 2017 at 3:48 PM Casey Stella <ceste...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > > Ok, so the problem here is that your profile is returning
> > integers
> > > >> > > (specifically HLLP cardinalities) rather than stats objects.
> When
> > > >> you're
> > > >> > > doing:
> > > >> > > STATS_PERCENTILE(STATS_MERGE( PROFILE_GET('host-talks-to',
> > > >> > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')), 90)
> > > >> > > You are calling STATS_MERGE on a list of integers and it takes a
> > > list
> > > >> of
> > > >> > > statistics objects.
> > > >> > >
> > > >> > > What you can do instead is:
> > > >> > > STATS_PERCENTILE( REDUCE( PROFILE_GET('host-talks-to',
> > > >> > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS'), (s, x) ->
> > STATS_ADD(s,
> > > >> x),
> > > >> > > STATS_INIT()), 90)
> > > >> > >
> > > >> > > Ok, that looks horrible, doesn't it? Well, thankfully we added
> > > >> temporary
> > > >> > > variables for stellar enrichments in 0.4.1. Let's take that
> > > "numeric"
> > > >> > > stellar enrichment group and reimagine it. With temporary
> > > variables,
> > > >> you
> > > >> > > would turn:
> > > >> > >
> > > >> > > "numeric" : {
> > > >> > > "value_red_level_out": "STATS_PERCENTILE( REDUCE(
> > > >> > > PROFILE_GET('host-being-talked-to', ip_src_addr,
> > PROFILE_FIXED(1,
> > > >> > > 'HOURS')), (s, x) -> STATS_ADD(s, x), STATS_INIT()), 95)",
> > > >> > > "value_red_level_in": "STATS_PERCENTILE( REDUCE(
> > > >> > > PROFILE_GET('host-talks-to',
> > > >> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS')), (s, x) -> STATS_ADD(s,
> > x),
> > > >> > > STATS_INIT()), 95)"
> > > >> > > },
> > > >> > >
> > > >> > > into:
> > > >> > > "numeric" : [
> > > >> > > "profile_duration := PROFILE_FIXED(1, 'HOURS')",
> > > >> > > "host_being_talked_to := PROFILE_GET('host-being-
> > > >> > talked-to',
> > > >> > > ip_src_addr, profile_duration)",
> > > >> > > "host_talks_to := PROFILE_GET('host-talks-to',
> > > >> ip_src_addr,
> > > >> > > profile_duration)",
> > > >> > > "host_being_talked_to_stats := REDUCE(
> > > >> host_being_talked_to,
> > > >> > > (s, x) -> STATS_ADD(s, x), STATS_INIT())",
> > > >> > > "host_talks_to_stats := REDUCE(host_talks_to, (s, x) ->
> > > >> > > STATS_ADD(s, x), STATS_INIT())",
> > > >> > > "value_red_level_out": "STATS_PERCENTILE(
> > > >> > > host_being_talked_to_stats, 95)",
> > > >> > > "value_red_level_in": "STATS_PERCENTILE(
> > > >> > host_talks_to_stats,
> > > >> > > 95)",
> > > >> > > "profile_duration := null",
> > > >> > > "host_being_talked_to := null",
> > > >> > > "host_talks_to := null",
> > > >> > > "host_being_talked_to_stats := null",
> > > >> > > host_talks_to_stats := null"
> > > >> > > ],
> > > >> > >
> > > >> > > That's a lot more to type, but it allows you to reuse and take
> > the
> > > >> pieces
> > > >> > > in chunks.
> > > >> > >
> > > >> > > Ok, so now I find myself thinking "a pox on both your houses"
> > since
> > > >> both
> > > >> > > examples now kinda look long and convoluted. So, why are they?
> > > Well,
> > > >> > that
> > > >> > > REDUCE is likely the culprit. It's supposed to get us out of bad
> > > >> > > situations not show up in what could be argued is the 80% case.
> > How
> > > >> > about,
> > > >> > > instead, we allow STATS_ADD or STATS_INIT to take a list of
> > > >> numbers? If
> > > >> > > so, we could pretty easily make that nicer:
> > > >> > > STATS_PERCENTILE( STATS_ADD( PROFILE_GET('host-being-
> > > talked-to',
> > > >> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
> > > >> > >
> > > >> > > or
> > > >> > > STATS_PERCENTILE( STATS_INIT(
> > > >> PROFILE_GET('host-being-talked-to',
> > > >> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
> > > >> > >
> > > >> > >
> > > >> > > We should make some sort of candy like that so we can avoid some
> > of
> > > >> the
> > > >> > > complexity in the normal case.
> > > >> > >
> > > >> > > On Wed, Aug 9, 2017 at 3:03 PM, Dima Kovalyov <
> > > >> dima.koval...@sstech.us>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hello Metron Team,
> > > >> > > >
> > > >> > > > I have created following profiler:
> > > >> > > > > {
> > > >> > > > > "profile": "host-talks-to",
> > > >> > > > > "onlyif": "exists(source_ip)",
> > > >> > > > > "foreach": "source_ip",
> > > >> > > > > "init": {
> > > >> > > > > "outcoming": "HLLP_INIT(5, 6)"
> > > >> > > > > },
> > > >> > > > > "update": { "outcoming": "HLLP_ADD(outcoming,
> > destination_ip)"
> > > >> },
> > > >> > > > > "result": "HLLP_CARDINALITY(outcoming)"
> > > >> > > > > }
> > > >> > > > I have also created enrichment rule:
> > > >> > > > > {
> > > >> > > > > "enrichment" : {
> > > >> > > > > "fieldMap": {
> > > >> > > > > "stellar" : {
> > > >> > > > > "config" : {
> > > >> > > > > "numeric" : {
> > > >> > > > > "value_red_level_out": "STATS_PERCENTILE(
> > > STATS_MERGE(
> > > >> > > > > PROFILE_GET('host-being-talked-to', ip_src_addr, 1,
> > 'HOURS')),
> > > >> 95)",
> > > >> > > > > "value_red_level_in": "STATS_PERCENTILE(
> > > STATS_MERGE(
> > > >> > > > > PROFILE_GET('host-talks-to', ip_src_addr, 1, 'HOURS')), 95)"
> > > >> > > > > },
> > > >> > > > > "text" : {
> > > >> > > > > "is_alert": "true"
> > > >> > > > > }
> > > >> > > > > }
> > > >> > > > > }
> > > >> > > > > }
> > > >> > > > > } }
> > > >> > > > However when I stream data to it I receive:
> > "value_red_level_out":
> > > >> > null,
> > > >> > > >
> > > >> > > > I have checked in profiler client and here is what I got:
> > > >> > > > > [Stellar]>>> PROFILE_GET( "host-talks-to" ,
> "99.191.183.156",
> > > >> > > > > PROFILE_FIXED(300, "MINUTES"))
> > > >> > > > > [1, 6, 6, 6, 6, 6, 3, 4, 5, 6, 4, 6, 6, 6, 1, 1, 6, 6, 1, 4,
> > 1,
> > > >> 1, 4,
> > > >> > > > > 6, 6, 1, 6, 6, 1, 2, 6, 1, 1, 1, 6, 4, 6, 6, 3, 1, 6, 2, 1,
> > 6,
> > > 1,
> > > >> 6]
> > > >> > > > > [Stellar]>>> STATS_PERCENTILE(STATS_MERGE(
> > > >> > > > > PROFILE_GET('host-talks-to', '99.191.183.156',
> > PROFILE_FIXED(10,
> > > >> > > > > 'HOURS'))), 90)
> > > >> > > > > NaN
> > > >> > > > > [Stellar]>>> STATS_MERGE( PROFILE_GET('host-talks-to',
> > > >> > > > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')))
> > > >> > > > So the STATS_MERGE produces no results. Is this something
> > expected
> > > >> or I
> > > >> > > > made a mistake somewhere? Please advise.
> > > >> > > >
> > > >> > > >
> > > >> > > > p.s. I am following this use cases:
> > > >> > > >
> > > >> > > https://github.com/hortonworks-gallery/metron-
> > > >> > rules/tree/master/use-cases/
> > > >> > > > DegreeOfHost
> > > >> > > > There were number of errors in the configs originally, which I
> > > have
> > > >> > > > corrected, maybe I missed something else.
> > > >> > > >
> > > >> > > > - Dima
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
> >
>

Reply via email to