[
https://issues.apache.org/jira/browse/METRON-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366666#comment-15366666
]
Casey Stella commented on METRON-265:
-------------------------------------
Well, I'm a fan of getting out of the way of the user, but I see model
discovery and deployment a compelling enough feature that is sufficiently
difficult to do at scale that it's worth-while being included. If we want to
just claim that supporting a REST adapter is sufficient, then we can do that
via another JIRA. The reason why I submitted this instead of that is that I
think it's sufficiently challenging to do that "just spin up a REST service" is
insufficient. It also ignores a lot of the benefits and tooling that we have
within Hadoop:
* Using Yarn to manage containers where the endpoints lie even within a
heterogeneous cluster hardware-wise
* Using zookeeper for auto discovery so the endpoint doesn't have to be made
explicit in configuration
I guess my point is that we can and should take an extra step toward
infrastructure to run this kind of service. As it stands, you are right,
though, anyone can create an enrichment adapter to call to a REST service. I
just think we can do better.
> Provide Model as a Service infrastructure to Metron
> ---------------------------------------------------
>
> Key: METRON-265
> URL: https://issues.apache.org/jira/browse/METRON-265
> Project: Metron
> Issue Type: New Feature
> Reporter: Casey Stella
> Assignee: Casey Stella
> Fix For: 0.2.1BETA
>
> Attachments: Model Management Infrastructure in Metron.docx
>
>
> One of the main features envisioned and requested is the ability to augment
> the threat intelligence and enrichment processes with insights derived from
> machine learning or statistical models. The challenges with this sort of
> infrastructure are
> • Applying the model may be sufficiently computationally/resource
> intensive that we need to support scaling via load balancing, which will
> require service discovery and management.
> • Models require out of band and frequent training to react to growing
> threats and new patterns that emerge.
> • Models should be language/environment agnostic as much as possible.
> These should include small-data and big-data libraries and languages.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
