[
https://issues.apache.org/jira/browse/METRON-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366650#comment-15366650
]
Nick Allen commented on METRON-265:
-----------------------------------
I'd like to keep this as simple and petite as possible. In my mind, right now
Metron should not care how models are updated, how models are trained, how they
are stored, etc. All we really need is a simple mechanism to score a message.
To 'score a message' just involves 3 steps. (1) Provide the JSON message as
input to the model. (2) Allow the model to score the message. (3) Retrieve
the model output and attach it to the original JSON message.
Now to me this sounds exactly like enrichment. Why can we not just use our
existing enrichment functionality to allow models to score messages? If there
are gaps, then we just need to enhance the enrichment functionality to fill
those gaps.
For example, if we want to support a REST call to an external model service,
then all we have to do is provide an implementation of EnrichmentAdapter that
is able to reach out to a REST endpoint. If you want caching or to support
Protobuf or whatever else, you just need an EnrichmentAdapter that can do that.
Taking this approach allows us to sidestep some of the finer implementation
points that are being discussed on the mailing list now. Those finer points
are likely going to differ based on the environment and model in use. We don't
need to solve all those problems now. All we need is a means to solve them in
the future.
> Provide Model as a Service infrastructure to Metron
> ---------------------------------------------------
>
> Key: METRON-265
> URL: https://issues.apache.org/jira/browse/METRON-265
> Project: Metron
> Issue Type: New Feature
> Reporter: Casey Stella
> Assignee: Casey Stella
> Fix For: 0.2.1BETA
>
> Attachments: Model Management Infrastructure in Metron.docx
>
>
> One of the main features envisioned and requested is the ability to augment
> the threat intelligence and enrichment processes with insights derived from
> machine learning or statistical models. The challenges with this sort of
> infrastructure are
> • Applying the model may be sufficiently computationally/resource
> intensive that we need to support scaling via load balancing, which will
> require service discovery and management.
> • Models require out of band and frequent training to react to growing
> threats and new patterns that emerge.
> • Models should be language/environment agnostic as much as possible.
> These should include small-data and big-data libraries and languages.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
